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EXECUTIVE  SUMMARY 


A  research  project  to  examine  mariner  assessment 

The  assessment  of  mariner  proficiencies  by  practical  demonstration  is  mandated  by  the 
International  Maritime  Organization  (IMO)  in  its  1995  amendments  to  the  International 
Convention  on  Standards  of  Training,  Certification  and  Watchkeeping  for  Seafarers  (STCW 
Convention)  and  the  accompanying  Seafarer’s  Training,  Certification  and  Watchkeeping  Code 
(STCW  Code).  This  requirement  is  a  substantial  departure  from  earlier  practice,  and  methods  for 
developing,  conducting,  and  documenting  such  assessments  must  be  established.  The  United 
States  Coast  Guard’s  (USCG)  National  Maritime  Center  (NMC)  sponsored  a  research  project 
through  the  USCG  Research  and  Development  Center  (R&DC)  to  examine  the  implications  of 
the  mandate  and  to  ensure  that  the  best  practices  are  available  to  the  industry.  The  major 
objective  of  the  project  was  to  provide  a  systematic,  step-by-step  process  for  developing  reliable 
and  valid  assessments  and  to  investigate  its  feasibility  for  implementation. 

Project  efforts  in  this  and  subsequent  phases 

The  efforts  described  in  the  present  report  were  the  first  phase  of  the  project.  A  team,  consisting 
of  researchers  with  expertise  in  training  and  assessment  and  marine  educators,  reviewed  the 
requirements  of  the  STCW  Code,  relevant  USCG  Navigation  and  Vessel  Inspection  Circulars 
(NVICs),  Instructional  Systems  Development  (ISD)  literature,  and  the  best  practices  of  industry. 
From  these  sources,  they  distilled  a  systematic,  step-by-step  method  for  developing  assessments. 
To  test  and  refine  the  method  and  to  provide  a  sample  assessment,  they  applied  it  to  a  skill  that  is 
emphasized  in  the  STCW  Code,  proficiency  in  the  operational  use  of  Automatic  Radar  Plotting 
Aids  (ARP A).  Their  ARPA  assessment  procedure  was  designed  for  use  on  the  United  States 
Merchant  Marine  Academy’s  (USMMA)  high-fidelity,  real-equipment-based  ARPA  laboratory. 
The  development  methodology  was  successful,  as  documented  herein. 

This  report  first  describes  five  steps  to  developing  an  assessment  procedure:  identify  assessment 
objectives,  determine  assessment  methods,  specify  the  assessment  conditions,  define  proficiency 
criteria,  and  prepare  assessment  objectives.  This  process  is  illustrated  by  following  the  research 
team  members  during  their  development  of  their  ARPA  procedure.  Appendices  contain  the 
ARPA  assessment  objectives,  the  needed  simulator  exercises,  instructions  to  the  assessor  and  the 
candidate,  and  worksheets.  Sufficient  detail  is  provided  to  allow  others  to  reproduce  their 
procedures. 

During  this  first  phase  of  the  project,  the  team  identified  several  critical  aspects  of  mariner 
assessment  that  merited  further  examination  in  subsequent  phases.  These  additional  aspects  were 
examined  in  further  investigations.  The  second  phase  involved  the  design  of  a  systematic 
method  to  assist  an  instructor/assessor  in  evaluating  the  bewildering  variety  of  available 
simulators,  including  personal  computer  (PC)  based  simulators,  in  their  capability  to  support 
mariner  assessments.  This  method  is  described  in  another  project  report  titled  “Evaluating 
Simulators  for  Assessments  of  Mariner  Proficiency.”  Step-by-step  support  materials  for  a  marine 
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instructor  in  developing  valid  and  reliable  assessments  without  the  assistance  of  training 
consultants  were  prepared  with  the  participation  of  faculty  from  two  state  academies.  The 
resulting  workshop  materials,  a  developer’s  manual,  and  illustrative  assessment  procedures  for 
deck  and  engine  room  are  presented  in  a  third  report  titled  “A  Method  for  Developing  Mariner 
Assessments.”  Lastly,  researchers  worked  with  a  major  shipping  company  to  identify  the  special 
requirements  oi  conducting  over-the-shoulder  assessments  during  commercial  operations.  The 
shipboard  trials  are  described  in  a  fourth  report  titled  “Conducting  Mariner  Assessments,”  which 
includes  an  assessor’s  manual  for  the  shipboard  officer  and  sample  assessments  adapted  for 
shipboard  use. 


Recommendations  for  Implementation  of  Project  Findings  and 
Products 

For  the  most  effective  implementation  of  the  project’s  findings  on  performance-based  assessment 

and  its  supporting  products,  the  following  USCG  and  industry  actions  are  recommended: 

•  The  USCG  should  encourage  the  maritime  industry  to  use  the  performance-based  assessment 
method,  as  described  in  the  project  reports  and  materials.  As  a  first  step,  the  materials  in  this 
report  should  be  made  widely  available,  on  the  USCG  STCW  website 
(http://www.uscg.mil/stcw)  and  through  the  National  Technical  Information  Service  (NTIS). 
When  the  industry  becomes  more  familiar  with  the  method,  it  can  serve  as  the  basis  for  a  new 
NVIC.  In  addition,  the  USCG  should  submit  the  supporting  materials  to  the  IMO 
subcommittee  on  Standards  for  Training  and  Watchkeeping  and  recommend  their  further 
distribution. 

•  The  USCG  should  encourage  further  familiarity  with  the  performance-based  assessment 
approach  by  USCG  staff,  as  a  tool  for  discussions  of  assessment  with  the  industry  and  for  the 
review  of  procedures  submitted  for  approval. 

•  The  USCG  should  encourage  the  review  and  use  of  the  performance-based  assessment 
method  and  materials  by  those  groups  who  are  dealing  with  the  important  technical  issues  of 
assessment.  These  groups  include  the  Maritime  Academy  Simulator  Committee  (MASC), 
the  Merchant  Marine  Personnel  Advisory  Committee  (MERPAC),  and  academy  committees 
appointed  by  the  Maritime  Administration  (MARAD)  to  address  STCW  issues.  The 
materials  can  provide  a  common  approach  and  a  common  basis  for  discussion. 

•  Those  in  the  industry  who  are  responsible  for  training  and  assessment  of  mariner  proficiency 
in  academies,  training  schools,  and  shipping  companies  should  make  use  of  the  performance- 
based  assessment  method  and  the  materials  presented  here  as  a  guide  for  their  own 
development  of  assessment  procedures.  All  the  project  materials  are  especially  appropriate 
for  inclusion  in  train-the-trainer  courses.  The  developer’s  manual  and  the  assessor’s  manual 
are  also  appropriate  for  those  who  do  not  take  such  courses. 
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INTRODUCTION 


In  recent  years,  larger  ships,  multinational  crews,  and  more  complex  automation  have  increased 
the  demand  for  well-trained,  competent  mariners.  The  1995  amendments  to  the  International 
Maritime  Organization  (IMO)  International  Convention  on  Standards  of  Training,  Certification 
and  Watchkeeping  for  Seafarers  (STCW  Convention)  (IMO,  1996A)  and  its  annexed  Seafarers’ 
Training,  Certification  and  Watchkeeping  Code  (STCW  Code)  (IMO,  1996B)  recognized  these 
challenges  and  redefined  the  requirements  for  mariner  assessment.  The  STCW  Code  guidance 
regarding  assessment  practices  and  standards  emphasizes  the  need  for  an  assessment  of  mariner 
skills  and  knowledge  that  incorporates  a  practical  demonstration.  In  defining  this  term,  the 
United  States  Coast  Guard  (USCG)  provided  the  following  guidance  in  regulations  46  CFR 
10.103  and  46  CFR  12.01-6: 

Practical  demonstration  means  the  performance  of  an  activity  under  the  direct 
observation  of  a  designated  examiner  for  the  purpose  of  establishing  that  the 
performer  is  sufficiently  proficient  in  a  practical  skill  to  meet  a  specified  standard 
of  competence  or  other  objective  criterion. 

Thus,  the  USCG  established  policy  in  support  of  a  shift  in  assessment  toward  greater  use  of 
practical  demonstrations  (46  CFR  10.103,  1997;  46  CFR  12.01-6,  1997;  USCG,  1997A;  USCG 
1997B;  USCG  1997C).  The  shift  toward  a  wider  application  of  performance-based  assessment 
in  the  maritime  industry  requires  the  development  of  a  broad  spectrum  of  assessment  procedures 
that  specify  how  assessments  are  to  be  conducted  and  what  objective  criteria  will  be  used  to 
determine  mariner  proficiency.  In  order  to  meet  this  requirement,  methods  for  developing, 
conducting,  and  documenting  such  assessments  are  needed.  The  USCG  National  Maritime 
Center  (NMC)  sponsored  a  research  project  through  the  USCG  Research  and  Development 
Center  (R&DC)  to  examine  the  implications  of  these  requirements  and  to  ensure  that  the  best 
practices  of  assessment  are  available  to  the  maritime  industry. 

On  this  project,  research  staff  worked  with  representatives  of  the  U.S.  maritime  industry  over 
several  years,  examining  the  STCW  Code  requirements,  training  and  assessment  methods  in  the 
literature,  and  the  best  existing  assessment  practices.  Together,  they  developed  STCW-  and 
USCG-compliant  methods,  tested  them  in  practical  settings,  and  revised  them  to  better  support 
the  industry  in  assessing  mariner  performance.  The  present  report  describes  the  technical  basis 
of  the  project’s  approach  to  performance-based  assessment,  which  was  prepared  by  a  team  of 
researchers  working  with  faculty  from  the  U.S.  Merchant  Marine  Academy  (USMMA).  As  they 
prepared  a  general  approach,  they  exercised  and  refined  it  with  a  specific  application  to 
assessment  of  mariner  proficiency  in  the  use  of  Automatic  Radar  Plotting  Aids  (ARPA).  They 
designed  the  approach  using  the  USMMA’s  high-fidelity  radar  simulator  laboratory  as  a  testbed. 
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During  the  first  phase  of  the  project,  which  is  reported  here,  the  team  identified  several  critical 
aspects  of  mariner  assessment  that  merited  further  examination.  These  other  aspects  were 
examined  in  further  investigations.  A  systematic  method  to  assist  an  instructor/assessor  in 
evaluating  the  bewildering  variety  of  available  simulators  in  their  capability  to  support  mariner 
assessments  is  described  in  another  project  report  (Raby,  Forsythe,  McCallum,  &  Smith,  2000). 
Step-by-step  support  material  for  a  marine  instructor  in  developing  valid  and  reliable  assessments 
without  the  assistance  of  training  consultants  was  prepared  with  the  participation  of  faculty  from 
the  Massachusetts  Maritime  Academy  (MMA)  and  the  California  Maritime  Academy  (CMA). 
The  resulting  workshop  materials,  a  developer’s  manual,  and  illustrative  assessment  procedures 
for  deck  and  engine  room  are  presented  in  a  third  report  (McCallum,  Forsythe,  Barnes,  Smith, 
Macaulay,  Sandberg,  Murphy,  &  Jackson,  2000).  Lastly,  researchers  worked  with  SeaRiver 
Maritime,  Inc.  (SRM)  to  identify  the  special  requirements  of  conducting  over-the-shoulder 
assessments  during  commercial  operations.  The  shipboard  trials  are  described  in  a  fourth  report 
(McCallum,  Barnes,  Forsythe,  Smith,  Maynard,  Blanchard,  Hempstead,  Martinez,  Murphy,  & 
Jackson,  2000),  which  includes  an  assessor’s  manual  for  the  shipboard  officer  and  sample 
assessments  adapted  for  shipboard  use. 


Objectives  and  Approach  of  the  Present  Study  Phase 

The  purpose  of  the  present  study  phase  was  to  explore  the  requirements  for  developing  and 
documenting  performance-based  assessments  that  would  conform  to  the  STCW  Code  guidance 
regarding  assessments  based  on  practical  demonstrations.  This  exploration  began  with  a  review 
of  the  Instructional  Systems  Development  (ISD)  literature  and  past  work  in  mariner  assessment. 
Based  on  this  review,  a  general  method  for  developing  performance-based  assessments  was 
prepared.  Then,  a  trial  application  of  the  method  was  conducted  for  the  purposes  of  evaluating 
and  refining  the  method,  as  well  as  to  develop  an  illustrative  assessment  procedure. 

The  trial  application  had  the  explicit  purpose  of  meeting  the  STCW  Code  guidance  regarding 
assessment  through  mariner  demonstration.  Because  the  requirements  for  conducting  an 
assessment  of  Automatic  Radar  Plotting  Aid  (ARP A)  operations  are  well  defined  in  the  STCW 
Code,  it  was  selected  as  the  proficiency  for  this  first  practical  application  of  the  assessment 
development  method.  In  addition,  the  study  team  began  by  applying  the  method  in  the  more 
controlled  environment  of  a  maritime  training  institution  before  addressing  the  more  diverse 
requirements  of  the  commercial  maritime  industry. 

In  exploring  the  requirements  for  developing  performance-based  assessments,  two  study 
objectives  were  established: 

•  Specify  and  refine  a  method  for  developing  performance-based  assessments  of  mariner  skills 
and  knowledge  that  is  responsive  to  IMO  and  USCG  guidance. 

•  Develop  a  performance-based  assessment  for  ARPA  operation  that  can  serve  as  one  example 
of  an  assessment  procedure  that  conforms  to  the  standards  of  competence  in  the  STCW  Code. 
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Report  Organization 

The  remainder  of  the  main  body  of  this  report  is  divided  into  three  sections.  First,  a  general 
method,  derived  from  a  review  of  the  ISD  literature,  is  presented  for  developing  performance- 
based  assessments  of  mariner  proficiency.  Next,  the  trial  application  of  this  method  to  the 
development  of  an  ARPA  operator  assessment  procedure  is  described.  Finally,  conclusions 
regarding  this  STCW-compliant  assessment  development  method  and  recommendations 
regarding  its  implementation  are  presented. 

Following  the  main  body  of  this  report  is  a  list  of  referenced  documents  and  a  glossary  of  the 
proficiency  assessment  terms  used  in  this  report.  Lastly,  a  series  of  appendices  document  the 
ARPA  operator  assessment  procedure  and  its  development.  Appendix  A  provides  the  assessment 
objectives,  conditions,  and  performance  measures.  Appendix  B  documents  the  ARPA  simulator 
assessment  exercises.  Appendix  C  documents  the  assessment  candidate  instructions  and 
worksheets.  Appendix  D  documents  the  assessor’s  instructions. 
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A  GENERAL  METHOD  FOR  DEVELOPING  PERFORMANCE-BASED 
ASSESSMENTS  OF  MARINER  PROFICIENCY 


Structured  approaches  for  developing  assessment  procedures  based  on  practical  demonstration 
have  been  available  for  several  decades  as  part  of  broader  methods  aimed  at  the  development  of 
performance-based  training  programs.  The  precursor  to  many  of  the  performance-based  training 
methods  used  in  the  United  States  is  commonly  referred  to  as  the  Instructional  Systems 
Development  (ISD)  method.  This  method  was  originally  developed  for  application  within  the 
U.S.  Department  of  Defense  (USDOD)  and  has  been  adopted  and  implemented  throughout  the 
U.S.  military  services  (Meister,  1985).  The  ISD  method  has  been  highly  successful  in  providing 
a  common  structure  for  developing  training  and  assessment  programs  for  a  wide  range  of  military 
skills  and  duties.  However,  due  to  the  complex  and  specialized  nature  of  the  techniques 
involved,  it  has  had  limited  application  outside  of  the  USDOD  environment. 

The  method  for  developing  performance-based  assessments  of  mariner  proficiencies  presented  in 
this  report  has  been  distilled  from  the  original  ISD  method  (TRADOC,  1977),  as  well  as  a 
number  of  more  recent  developments  in  instructional  development  (Gagne,  Briggs,  &  Wagner, 
1992;  Goldstein,  1993;  Jonassen,  Hannum,  &  Tessmer,  1989;  Krieger,  1994;  Mager,  1997; 
Rothwell  &  Kazanas,  1998;  Sanquist,  Lee,  &  Rothblum,  1994;  Smith  et  al.,  1996;  and  Sullivan  & 
Elenburg,  1988).  The  method  proposed  here  intends  to  preserve  the  ISD  objectives  of 
establishing  valid  and  reliable  performance  objectives,  measures,  and  standards.  However,  the 
present  method  also  represents  a  significantly  streamlined  version  of  ISD  that  is  intended  to 
better  serve  the  operational  constraints  within  the  U.S.  maritime  industry.  The  current  method  is 
targeted  at  users  in  a  broad  range  of  maritime  organizations,  including  maritime  academies  that 
provide  four-year  degrees,  maritime  colleges  with  two-year  programs,  commercial  maritime 
training  institutions  that  provide  integrated  curricula  or  individual  courses,  and  shipping 
companies  that  develop  in-house  training  and  certification  programs. 

The  step-by-step  method  is  described  in  this  section  and  illustrated  in  the  next  section  by  its 
application  to  an  assessment  of  ARPA  proficiency  developed  by  the  researchers  and  the 
USMMA  faculty.  Note  that  another  treatment  of  these  steps  is  presented  in  a  “developer’s 
manual”  for  the  instructor/assessor  in  another  project  report  (McCallum,  Fors)4he,  Bames, 

Smith,  Macaulay,  Sandberg,  Murphy,  &  Jackson,  2000).  The  method  is  consistent  with  the 
guidance  in  a  Circular  on  assessment  published  by  the  International  Maritime  Organization 
(1998). 
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Method  Overview 


Figure  1  depicts  the  proposed  five-step  method  for  developing  performance-based  assessments  of 
mariner  proficiency.  The  method  consists  of  a  series  of  four  analytic  steps,  followed  by  a  final 
step  for  the  preparation  of  the  assessment  package.  The  first  step  in  the  method  is  to  identify  the 
assessment  objectives,  which  are  the  various  aspects  of  mariner  proficiency  that  are  to  be 
assessed.  The  second  step  is  to  determine  the  assessment  method  (or  methods)  to  be  used  in 
conducting  the  assessments.  The  third  step  calls  for  specifying  the  assessment  conditions,  which 
are  the  conditions  necessary  for  eliciting  the  mariner’s  performance.  The  fourth  step  is  to  define 
the  criteria  that  are  to  serve  as  objective  indicators  of  mariner  proficiency.  Finally,  in  the  fifth 
step,  the  results  of  the  preceding  analyses  are  compiled  into  a  documented  assessment  procedure. 
The  remainder  of  this  section  provides  a  general  description  of  each  step  in  the  performance- 
based  assessment  development  method,  along  with  a  discussion  of  some  issues  that  should  be 
addressed  in  conducting  each  step. 


Step  1 :  Specify  Assessment  Objectives 


Figure  1.  Five-step  method  for  developing  mariner  performance-based  assessments. 

Step  1:  Specify  Assessment  Objectives 

The  first  step  in  developing  a  performance-based  assessment  is  to  identify  the  assessment 
objectives.  The  fundamental  goal  in  identifying  assessment  objectives  is  to  specify  the  critical 
requirements  of  job  performance  that  can  be  measured  and  assessed  in  a  performance -based 
assessment.  These  requirements  take  two  general  forms.  First,  there  are  skills,  which  refer  to 
learned  behaviors  that  must  be  applied  in  successful  job  performance.  Second,  there  is 
knowledge,  which  refers  to  the  learned  concepts,  cues,  and  procedures  that  allow  the  mariner  to 
determine  when  and  how  to  apply  his  or  her  skills.  A  comprehensive  set  of  performance-based 
assessment  objectives  addresses  both  the  skill  and  knowledge  requirements  of  the  job.  This 
results  in  objectives  that  require  the  demonstration  of:  (1)  prerequisite  knowledge;  (2) 
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fundamental  job  skills;  and  (3)  ability  to  select  and  execute  skilled  performance,  based  on  the 
mariner’s  reference  to  his  or  her  knowledge. 

Assessment  objectives  should  be  based  on  a  detailed  understanding  of  the  job  requirements 
corresponding  to  the  proficiency  of  interest.  For  assessments  that  are  to  be  developed  in 
accordance  with  proficiencies  identified  in  the  STCW  Code,  that  document  should  serve  as  the 
starting  point  for  this  review.  The  “Standard  of  Competence”  tables  in  the  STCW  Code  specify, 
in  general  terms,  four  sets  of  information  intended  to  guide  the  assessment  development  process: 

•  The  competence,  or  broadly  defined  job  requirements. 

•  The  knowledge,  understanding  and  proficiency  requirements  associated  with  the  competence. 

•  Methods  for  demonstrating  competence. 

•  Criteria  for  assessing  competence. 

To  ensure  compliance  with  these  requirements,  the  assessment  developer  must  carefully  review 
the  information  in  the  STCW  Code  tables  that  correspond  to  the  area  of  assessment.  However,  a 
review  of  these  tables  reveals  that  the  mariner  skill  and  knowledge  requirements  corresponding 
to  the  proficiency  must  be  further  analyzed  to  specify  the  associated  assessment  objectives. 
Following  an  initial  review  and  analysis  of  the  appropriate  portions  of  the  STCW  Code,  an 
assessment  developer  should  next  review  any  additional  requirements  pertinent  to  the  proficiency 
area  under  analysis.  In  the  United  States.,  the  primary  source  for  regulatory  requirements  is 
found  in  the  Code  of  Federal  Regulations  (CFR),  Title  46  -  Shipping.  Other  requirements  for  a 
given  job  may  also  be  found  in  union  agreements  and  company  operating  procedures.  Following 
review  of  the  regulatory  requirements,  assessment  developers  may  need  to  refer  to  technical 
manuals,  job  instructions,  textbooks,  and  task  analyses  to  aid  in  the  identification  of  job 
requirements. 

In  most  cases,  an  individual  proficiency  is  analyzed  to  identify  a  set  of  several  related  assessment 
objectives  that  should  be  addressed  separately  during  mariner  proficiency  assessment.  Each  of 
these  objectives  should  represent  a  critical  job  requirement  corresponding  to  the  proficiency 
being  analyzed.  There  are  numerous  job  analysis  methods  that  can  be  used  in  defining 
assessment  objectives.  At  this  point,  broad  areas  of  knowledge,  skill,  and  ability  that  are  critical 
to  successful  job  performance  should  be  identified.  Later,  during  development  of  the  proficiency 
criteria,  a  more  detailed  task  analysis  should  be  conducted.  Further  guidance  on  the  conduct  of 
job  and  task  analyses  can  be  found  in  texts  that  address  these  topics  in  detail  (Janassen,  Hannum, 
&  Tessmer,  1989;  Mager,  1997;  Rothwell  &  Kazanas,  1998). 

An  individual  assessment  procedure  should  address  either  a  single  proficiency,  or  an  integrated 
set  of  proficiencies  defined  in  the  STCW  Code.  In  addition,  the  time  and  resource  requirements 
for  assessment  should  be  within  a  practical  level.  An  assessment  is  too  broad  in  scope  if  it 
addresses  unrelated  proficiencies  or  if  its  conduct  requires  an  unnecessarily  extensive  amount  of 
time  and  resources. 
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To  ensure  consistency  and  completeness  when  specifying  assessment  objectives,  a  standard 
format  should  be  applied.  Each  assessment  objective  should  include  a  general  description  of  the 
mariner  actions  to  be  demonstrated  and  the  conditions  under  which  they  are  to  be  demonstrated. 
The  conditions  should  identify  any  special  characteristics  of  the  assessment  setting,  such  as  day 
or  night  conditions  and  major  types  of  equipment  or  equipment  settings  required  for  proficiency 
assessment. 

Following  specification  of  the  assessment  objectives,  it  is  often  appropriate  to  ensure  that  they 
are  valid.  A  valid  assessment  objective  is  one  that  accurately  reflects  the  skill,  knowledge,  and 
performance  requirements  of  the  job.  A  commonly  accepted  means  of  establishing  the  validity  of 
a  set  of  assessment  objectives  is  to  reach  consensus  on  the  objectives  among  a  group  of  subject 
matter  experts.  An  open  and  frank  discussion  of  the  knowledge,  skill,  and  performance 
requirements  corresponding  to  a  proficiency  provides  a  solid  basis  for  establishing  valid 
assessment  objectives. 

Step  2:  Determine  Assessment  Methods 

The  intent  of  the  STCW  Code  is  to  foster  assessment  through  practical  demonstration.  Thus,  it  is 
important  that  assessment  developers  carefully  consider  the  alternative  assessment  methods 
available  to  them  and  select  the  method  that  best  meets  the  intent  of  the  STCW  Code,  while  also 
considering  the  practical  limitations  of  mariner  assessment.  There  are  three  basic  methods  of 
assessment: 

•  Written  or  oral  questions. 

•  Exercises  in  a  simulated  job  setting. 

•  Exercises  in  an  actual  shipboard  job  setting. 

At  the  most  general  level,  there  are  two  basic  categories  of  what  is  being  assessed  by  an 
assessment  procedure.  First,  an  assessment  can  assess  the  candidate’s  demonstration  of  his  or 
her  knowledge  of  concepts,  constructs,  rules,  and  procedures.  Often,  specific  areas  of  knowledge 
can  be  identified  as  a  prerequisite  to  proficiency  in  job  performance.  In  such  cases,  assessment 
of  a  candidate’s  knowledge  is  appropriate.  Second,  an  assessment  can  assess  the  candidate’s 
demonstration  of  his  or  her  ability  to  apply  knowledge  and  skills  in  an  operational  setting.  This 
type  of  assessment  is  the  primary  focus  of  the  present  method.  Demonstration  of  ability  in 
operational  settings  provides  a  much  more  valid  basis  for  assessment.  However,  due  to 
economic,  efficiency,  or  safety  concerns,  sometimes  the  assessment  of  knowledge  demonstration 
in  a  non-operational  setting  is  appropriate. 

General  guidance  in  the  selection  of  assessment  methods  can  be  provided,  as  summarized  in 
Table  1.  As  presented  here,  written  or  oral  questions  are  used  to  assess  the  mariner’s 
demonstration  of  knowledge.  On  the  other  hand,  simulated  or  actual  shipboard  job  settings  are 
used  to  assess  the  mariner’s  demonstration  of  the  ability  to  apply  knowledge  and  skills  in  an 
operational  setting. 
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Table  1.  General  guidance  on 

the  selection  of  assessment  methods. 

What  is  Being  Assessed 

Method  of  Mariner  Assessment 

Knowledge  of  concepts,  constructs,  rules,  and 
procedures 

Oral  and  written  questions 

Demonstration  of  the  ability  to  apply  knowledge 
and  skills  in  an  operational  setting 

Exercises  In  real  or  simulated  operations 

Once  the  alternative  assessment  methods  have  been  identified,  the  assessment  developer  should 
consider  the  advantages  and  disadvantages  of  each  alternative.  Factors  to  consider  are 
assessment  validity,  assessment  reliability,  ease  of  assessment  development,  and  ease  of 
assessment  administration.  In  the  context  of  assessment,  validity  can  be  defined  as  the  degree  to 
which  knowledge,  skill,  and  performance  requirements  of  the  job  are  accurately  reflected  by 
assessment  requirements.  Reliability  can  be  defined  as  the  consistency  of  the  assessment 
outcome  following  administration  under  comparable  conditions.  Table  2  summarizes  some  of 
the  common  advantages  and  disadvantages  associated  with  each  of  the  three  general  assessment 
methods. 


Table  2.  Assessment  method  advantages  and  disadvantages. 


Method 

Advantage 

Disadvantage 

Written/Oral 

Question 

Easy  to  standardize 

Easy  to  assess  many  candidates 
Good  measure  of  knowledge 

May  not  adequately  represent  actual 
requirements  in  work  setting  (validity) 

Simulator 

Practical 

Demonstration 

High  reliability 

Safe 

May  have  high  validity 

May  not  adequately  represent  actual 
requirements  in  work  setting  (validity) 

May  be  expensive 

Usually  need  one  assessor  per  candidate 

Shipboard 

Practical 

Demonstration 

High  validity 

Usually  need  one  assessor  per  candidate 

More  difficult  to  control  conditions  (reliability) 

Task  may  be  infrequent  or  dangerous 

The  assessment  developer  should  consider  each  assessment  objeetive  and  define  an  assessment 
method  corresponding  to  each  objective.  The  definition  of  the  selected  assessment  method 
should  include  a  preliminary  description  of  the  assessment  method  type  and  assessment 
conditions  for  each  objective.  Assessment  methods  and  conditions  should  provide  a  reasonable 
level  of  operational  fidelity.  That  is,  they  should  be  as  faithful  as  possible  to  conditions  present 
in  an  actual  shipboard  environment.  Assessment  methods  should  also  provide  a  means  of 
maintaining  sufficient  control  over  events  so  that  mariner  safety  is  optimized  and  operational 
risks  are  minimized. 
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step  3:  Specify  Assessment  Conditions 

Following  the  determination  of  assessment  methods,  the  assessment  developer  should  begin  a 
more  detailed  description  of  how  the  assessment  will  be  conducted.  The  assessment  conditions 
provide  a  good  point  of  departure  for  this  detailed  specification.  The  primary  concern  in 
specifying  assessment  conditions  is  to  ensure  that  comparable  conditions  can  be  repeated  from 
one  application  to  the  next.  The  following  should  be  included  in  a  detailed  description  of  the 
assessment  conditions: 

•  Assessor  instructions. 

•  Candidate  instructions. 

•  Tools  and  apparatus  used  by  candidate. 

•  Written  and  oral  questions. 

•  Simulator  exercise  requirements. 

•  Shipboard  operations  requirements. 

The  assessor  instructions  should  address  the  general  procedures  and  requirements  for  conducting 
an  assessment.  In  addition,  special  assessment  requirements  that  the  assessor  needs  to  prepare  in 
advance  should  be  identified  in  the  assessor  instructions.  In  the  case  of  simulator-based 
assessments,  these  requirements  will  include  familiarization  with  the  controls,  displays,  and 
outputs  of  the  simulator.  In  the  case  of  shipboard  assessments,  these  requirements  will  include 
specific  operational  conditions  required  to  ensure  a  valid,  reliable,  and  safe  assessment. 

Candidates  should  be  provided  with  the  appropriate  information  to  allow  adequate  preparation 
for  assessment.  However,  these  instructions  should  not  be  a  self-contained  training  resource. 

The  instructions  should  identify  training  courses  or  resources  that  address  the  performance 
requirements  corresponding  to  the  assessment  objectives  under  consideration.  If  simulators  are 
being  employed,  a  detailed  description  of  the  initial  conditions  of  the  operational  scenario  should 
be  provided  in  the  candidate’s  instructions,  specifying  both  the  initial  settings  of  equipment,  as 
well  as  any  control  of  equipment  that  is  exercised  by  the  candidate  during  the  course  of  the 
assessment.  If  shipboard  assessment  is  being  used,  the  operations  under  which  the  assessment  is 
conducted  should  be  described  and  the  initial  settings  of  operational  equipment  should  be 
specified.  This  description  requires  comparable  detail  to  that  of  the  simulator  assessment, 
although  control  of  these  conditions  may  be  more  difficult. 

Tools  and  apparatus  used  by  the  candidate  typically  vary  between  simulator  and  shipboard 
assessments.  If  assessment  is  being  conducted  aboard  ship,  a  subset  of  the  tools  and  equipment 
available  in  the  workplace  are  usually  specified.  If  assessment  is  being  conducted  in  a  simulated 
setting,  facsimile  tools  and  equipment  are  often  developed  for  the  purposes  of  training  and 
assessment. 

Written  and  oral  questions  should  be  adequately  specified  to  ensure  comprehensive  and 
consistent  assessment.  The  specific  topics  to  be  addressed  by  the  set  of  questions  should  be 
adequately  defined  so  that  the  administration  of  any  assessment  addresses  a  common  set  of 
concepts,  constructs,  rules,  and  problem  solving  skills.  In  the  best  case,  a  library  of  test  items  is 
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provided,  along  with  procedures  for  sampling  from  this  library  for  any  assessment.  At  a 
minimum,  the  number  of  questions  should  be  identified,  and  the  topics  and  subtopics  for  subsets 
of  questions  should  be  identified.  In  addition,  the  type  of  question  (e.g.,  open-ended,  multiple- 
choice,  fill-in,  and  essay/discussion)  also  should  be  specified. 

Finally,  the  simulator  exercise  or  shipboard  operations  requirements  must  be  specified.  The  use 
of  simulators  typically  allows  a  high  level  of  control  in  defining  the  sequence  of  events  presented 
to  the  candidate  during  assessment.  This  high  degree  of  control  can  often  be  used  to  advantage  in 
establishing  valid  and  reliable  assessments  that  consistently  present  the  operational  conditions  of 
interest.  Shipboard  operations  can  vary  substantially  in  the  level  of  control  that  is  available, 
depending  on  the  area  of  assessment  and  phase  of  shipboard  operations.  A  great  deal  of 
consideration  is  necessary  in  specifying  shipboard  assessment  conditions  to  ensure  that  they 
consistently  reflect  those  job  requirements  that  correspond  to  the  assessment  objectives  under 
consideration. 

Step  4:  Develop  Proficiency  Criteria 

Up  to  this  point  in  the  assessment  development  method,  assessment  objectives  are  defined  that 
correspond  to  specific  proficiencies  included  in  the  STCW  Code,  other  regulatory  requirements, 
or  major  job  requirements.  However,  for  the  purpose  of  defining  performance  that  can  be 
objectively  assessed,  the  performance  required  by  an  assessment  objective  may  need  to  be  broken 
into  smaller,  more  discrete  components.  The  separate  components  of  an  objective  consist  of 
knowledge  and/or  application  of:  (1)  concepts  or  constructs;  (2)  rules  of  operation;  or  (3)  steps 
or  procedures  for  operation.  Textbooks,  course  materials,  training  experts,  regulations, 
organizational  policies,  technical  manuals,  ship  manuals,  and  operational  experts  should  be 
referred  to,  as  necessary,  to  ensure  the  definition  of  valid  assessment  objective  components.  As 
noted  in  the  earlier  discussion  of  assessment  objective  validity,  review  and  discussion  among  a 
panel  of  subject  matter  experts  is  advisable. 

Reliable  and  valid  assessment  requires  the  consistent  measurement  of  mariner  performance,  and 
the  consistent  application  of  relevant  proficiency  criteria.  Proficiency  criteria  are  the  scoring 
procedures  applied  in  determining  the  proficiency  of  a  candidate  on  the  basis  of  performance 
measures  and  performance  standards.  A  performance  measure  identifies  how  a  candidate’s 
performance  is  observed  and  recorded.  A  common  distinction  between  types  of  measures 
contrasts  objective  measures  and  subjective  measures.  An  objective  measure  can  be  defined  as 
one  that  relies  upon  measurement  apparatus  that  can  be  calibrated  to  yield  highly  consistent  and 
accurate  measurement  results.  Examples  of  apparatus  used  in  objective  measurement  are 
stopwatches,  compasses,  rudder  angle  indicators,  oil  pressure  gauges,  and  voltmeters.  An 
example  of  an  objective  performance  measure  is  the  time  required  to  launch  a  lifeboat.  This 
measure  would  include  specification  of:  (1)  how  to  signal  the  start  of  the  launching  assessment; 
(2)  when  to  start  the  timer;  (3)  which  timer  to  use  and  its  calibration;  and  (4)  when  to  stop  and 
record  the  launch  time.  Whenever  practical  and  meaningful,  objective  measures  should  be  used 
in  assessment  because  they  provide  the  most  reliable  means  of  measurement. 
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A  subjective  measure  can  be  defined  as  one  that  relies  primarily  on  the  assessor’s  observation 
and  interpretation  of  mariner  performance  to  determine  the  assessment  outcome.  Two  types  of 
subjective  measures  that  are  commonly  used  in  assessments  are  checklists  and  ratings.  A 
checklist  requires  the  assessor  to  observe  the  mariner’s  performance,  or  the  outcome  of 
performance,  and  choose  one  of  two  performance  levels,  typically  labeled  “pass”  and  “fail.”  A 
rating  requires  that  the  assessor  observe  the  mariner’s  performance,  or  the  outcome  of 
performance,  and  choose  one  of  three  or  more  performance  levels.  For  example,  a  three-level 
rating  could  assign  scores  of  0,  1,  and  2  to  the  observed  performance.  A  subjective  measure  is 
comprised  of  the  recording  form  and  the  instructions  for  how  to  categorize  the  observed 
performance.  Although  not  as  inherently  consistent  and  accurate  as  objective  measures,  the 
consistency  and  accuracy  of  subjective  measures  can  be  greatly  enhanced  by  carefully  defining 
what  performance  or  outcome  is  being  observed  and  how  it  should  be  categorized  and  recorded 
by  the  assessor. 

A  performance  standard  is  the  value  of  a  performance  measure  that  is  established  as  an 
acceptable,  or  target,  value.  In  the  case  of  objective  measures,  the  performance  standard  is 
typically  established  independently  of  the  measurement  procedures.  For  example,  the  time 
required  for  an  activity  to  be  performed  can  be  recorded,  then  compared  to  the  established 
standard  of  performance.  However,  in  the  case  of  subjective  measures,  the  performance  standard 
is  established  as  part  of  the  measurement  procedure.  For  oral  and  written  questions,  this  means 
the  inclusion  of  acceptable  answers.  When  the  subjective  measure  requires  the  assessor’s 
observation  of  procedural  steps  or  their  outcomes,  guidance  on  the  performance  that  is  to  be 
considered  proficient  must  be  included  with  the  assessor’s  recording  form.  In  developing 
standards  for  STCW  certification,  the  criterion  for  proficiency  specified  in  that  document  is  an 
important  first  source  for  establishing  standards.  Other  sources  are  the  same  ones  used  for 
establishing  the  assessment  objectives  and  breaking  them  into  components.  These  include 
government  regulations,  technical  manuals,  job  instructions,  textbooks,  task  analyses,  and  finally 
appropriate  experts. 

Table  3  provides  a  basic  set  of  guidelines  for  selecting  a  measure  type  and  developing 
performance  standards.  The  guidance  provided  in  the  second  and  third  columns  of  this  table 
assumes  prior  consideration  of  the  basis  of  performance  measurement,  identified  in  the  first 
column.  In  the  first  case,  if  performance  is  best  measured  on  the  basis  of  a  calibrated  apparatus, 
then  the  appropriate  measure  type  is  objective  and  the  performance  standard  should  be 
established  by  determining  an  acceptable  level  of  the  measure.  Second,  if  performance  is  best 
classified  into  two  pre-defined  levels,  then  the  appropriate  measure  type  is  a  subjective  checklist 
and  the  performance  standard  should  be  established  by  defining  the  two  levels  of  performance 
(typically  “pass”  and  “fail”).  Finally,  if  performance  is  best  classified  into  three  or  more  pre¬ 
defined  levels,  then  the  appropriate  measure  type  is  a  subjective  rating  and  the  performance 
standard  should  be  established  by  defining  each  of  the  levels  of  performance  (e.g.,  0,  1,  and  2). 
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Table  3.  Selecting  measure  types  and  establishing  performance  standards. 


IF  performance  is  best  measured  on 
the  basis  of... 

THEN  the  measure 
type  is... 

AND  the  performance  standard  should 
be  established  by... 

a  calibrated  apparatus 

objective 

determining  an  acceptable  level  of  the 
measure 

classification  into  two  pre-defined  levels 

subjective  checklist 

defining  the  two  levels  of  performance 

classification  into  three  or  more  pre¬ 
defined  levels 

subjective  rating 

defining  each  level  of  performance 

When  the  assessment  objectives  have  been  broken  down  into  individual  components  and 
performance  measures  and  standards,  it  is  typically  necessary  to  combine  these  separate  measures 
and  standards  into  a  more  integrated  metric  of  proficiency.  The  separate  measures  and  standards 
should  be  reviewed  with  respect  to  the  broader  assessment  objectives  and  requirements  for 
mariner  proficiency.  Based  on  this  review,  scoring  procedures  should  be  developed  that  combine 
individual  measures  to  establish  a  proficiency  criterion  for  each  larger  assessment  objective. 

The  proficiency  criteria  corresponding  to  each  assessment  objective  must  be  validated  in  a 
manner  that  ensures  that  assessment  will  represent  the  requirements  of  the  job.  Proficiency  is,  in 
the  end,  a  judgment  that  takes  into  account  the  risks  associated  with  sub-optimal  performance 
and  the  costs  of  requiring  optimal  performance  of  all  mariners.  A  final,  formal  review  of  the 
proficiency  criteria  by  a  panel  of  experts  typically  provides  justification  of  the  validity  of  an 
assessment. 

Step  5:  Prepare  the  Assessment  Package 

Following  the  completion  of  the  analytic  steps  that  define  the  content  of  an  assessment,  the 
assessment  procedures  should  be  documented  to  ensure  consistent  application  during  assessment. 
The  assessment  package  should  be  prepared  as  a  stand-alone  procedure.  That  is,  an  assessor  not 
familiar  with  the  history  of  its  development  should  be  able  to  administer  it.  The  goal  should  be 
the  preparation  of  clear  and  concise  instructions  that  can  be  used  by  multiple  assessors  in 
obtaining  valid  and  reliable  assessment  results  over  multiple  applications.  The  resulting 
assessment  package  should  include  six  major  components: 

•  Assessment  worksheets. 

•  Assessor  instructions. 

•  Candidate  instructions. 

•  Written  and  oral  items. 

•  Simulator  and/or  shipboard  procedures. 

•  References,  source  materials,  and  validation  documentation. 
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The  central  products  of  the  assessment  package  are  the  written  procedures  or  worksheets  that  will 
be  used  by  the  assessor  while  conducting  the  assessment.  It  might  be  most  efficient  to  prepare 
these  worksheets  first,  since  they  likely  will  be  referred  to  in  all  of  the  other  assessment  package 
materials.  An  assessment  worksheet  should  provide  sufficient  information  to  guide  the  assessor 
through  the  assessment  procedure  and  provide  a  means  of  recording  the  outcome  corresponding 
to  the  individual  assessment  objectives,  measures,  and  standards.  It  should  be  noted  that  detailed 
assessor  instructions  could  take  the  place  of  highly  elaborate  assessment  worksheets.  That  is, 
much  of  the  information  required  for  conducting  an  assessment  could  be  included  in  the  assessor 
instructions,  with  only  the  minimum  necessary  for  recording  the  results  of  the  assessment 
included  in  the  worksheet.  On  the  other  hand,  assessor  instructions  could  be  limited  to  the 
necessary  information,  with  more  detailed  guidance  provided  directly  on  the  worksheet.  Factors 
to  consider  in  selecting  the  detail  included  in  the  worksheet  and  instructions  include  the  amount 
of  assessment  training  available  to  assessors  and  the  amount  of  information  that  can  be 
reasonably  included  in  these  two  documents. 

Every  assessment  package  should  include  a  set  of  assessor  instructions.  In  many  cases,  detailed 
step-by-step  instructions  will  help  to  ensure  the  reliability  and  validity  of  the  resulting 
assessment.  Although  proficient  mariners  should  be  conducting  the  assessment,  there  can  be 
many  interpretations  of  how  an  assessment  is  to  be  conducted,  especially  with  regard  to  briefing 
and  instruction  of  the  candidate,  equipment  set-up,  and  timing  and  pace  of  assessment  activities. 
Assessor  instructions  could  include  the  following  topics; 

•  Assessment  preparation. 

•  Candidate  pre-briefing. 

•  Performance  observations  and  recording. 

•  Determination  of  assessment  outcome. 

•  Assessment  debriefing. 

The  assessment  package  should  also  include  separate,  self-contained  candidate  instructions  that 
can  be  provided  to  the  candidate  well  in  advance  of  the  time  that  the  assessment  is  conducted. 

The  candidate  instructions  should  include,  at  a  minimum: 

•  Assessment  objective  source  {STCW  Code  chapter,  table,  function,  competence,  and 
proficiency). 

•  Assessment  objective. 

•  Assessment  method. 

•  Conditions  under  which  the  assessment  will  be  conducted  (equipment  status,  underway 
versus  in  port,  etc.). 

•  Assessment  procedures  (who  will  be  administering  the  assessment  and  how  long  it  will  take). 

•  A  summary  of  the  measures  and  standards  that  will  be  used  to  determine  the  outcome  of  the 
assessment. 
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Written  items  should  be  documented  either  by  defining  detailed  question  objectives  or  by 
developing  a  library  of  questions.  The  assessment  developer  must  be  aware  of  the  dangers  of 
inadequate  specification  of  written  or  oral  questions.  Without  a  detailed  set  of  questions,  the 
performance  measures  are  not  defined  and  any  performance  standards  are  arbitrary.  At  a 
minimum,  the  specific  knowledge  areas  to  be  assessed  should  be  defined,  along  with  the  format 
of  the  assessment  items  (i.e.,  true/false,  multiple-choice,  short  answer,  and  essay/discussion),  and 
the  number  of  items  that  should  be  included  for  each  knowledge  area.  The  developer  should 
consider  providing  a  detailed  listing  of  specific  questions,  along  with  the  correct  answers  for  each 
question. 

Just  as  it  is  necessary  to  detail  written  and  oral  questions,  the  detailed  procedures  for  simulator 
and/or  shipboard  assessment  procedures  should  also  be  finalized  in  the  assessment  package. 
When  simulators  are  used,  specific  assessment  exercises  or  scripts  should  be  developed  and 
documented.  Fully  documented  simulator  scripts  or  scenarios  might  include  the  following 
specifications: 

•  Simulated  time  of  day  and  weather  conditions. 

•  Own  vessel  status  at  the  start  of  the  scenario. 

•  Other  vessel  locations  at  the  start  of  the  scenario. 

•  Sequence  and  timing  of  own  ship  malfunctions,  if  any. 

•  Course,  speed,  and  maneuvers  of  other  vessels,  if  any. 

Specifying  shipboard  procedures  for  assessment  must  take  numerous  factors  into  consideration, 
as  noted  in  the  earlier  discussion  of  assessment  conditions.  Such  procedures  must  provide 
adequate  guidance  to  ensure  safe,  valid,  and  reliable  assessment,  while  recognizing  the  practical 
constraints  of  commercial  operations.  It  is  anticipated  that  selected  mariner  proficiencies  will  be 
identified  as  amenable  to  shipboard  assessment.  These  proficiencies  should  correspond  to  job 
activities  and  settings  that  can  be  conducted  with  relatively  high  levels  of  safety  and  control  over 
operational  conditions.  Further  research  and  development  work  is  required  in  this  area  to 
determine  the  appropriate  issues  that  must  be  addressed  to  ensure  valid  and  reliable  shipboard 
assessment. 

The  final  step  in  preparing  the  assessment  package  is  to  document  references,  source  materials, 
and  the  validation  process.  This  last  step  serves  two  purposes.  First,  it  provides  the  necessary 
references  and  documentation  to  allow  the  assessment  developer,  or  his/her  successor,  to 
maintain  and  update  the  assessment  procedure.  Second,  this  documentation  provides  reference 
material  that  can  be  used  by  an  external  reviewer  who  may  evaluate  the  assessment  procedure. 
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DEVELOPMENT  OF  AUTOMATIC  RADAR  PLOTTING  AID  OPERATOR 
ASSESSMENT  PROCEDURE 


The  basic  assessment  development  method  described  in  the  preceding  section  was  applied  in  an 
exploratory  effort  to  ARPA  operator  proficiencies.  This  trial  application  was  conducted  during 
the  fall  and  winter  of  1996  and  1997  by  a  team  consisting  of  USCG  R&D  Center  technical  staff, 
Battelle  technical  staff,  and  faculty  from  the  USMMA.  The  academy’s  ARPA  simulator 
laboratory  facilities  were  used  as  the  developmental  testbed.  The  following  discussion  is 
organized  around  the  five  steps  of  the  proposed  assessment  development  method.  For  each 
assessment  development  step,  identified  issues,  findings,  and  assessment  products  are  discussed. 

Assessment  Objectives 

The  specification  of  assessment  objectives  began  with  an  initial  review  of  several  source 
documents.  The  summary  of  competencies  for  ARPA  assessment  provided  in  the  STCW  Code 
(IMO,  1996B,  Section  B-I/12)  served  as  the  starting  point  for  this  review.  Further  detail  was 
obtained  from  a  review  of  two  additional  IMO  documents.  Model  Course  3.12  —  Examination 
and  Certification  of  Seafarers  (IMO,  1992)  and  Model  Training  Record  Book  for  Candidates  for 
Certification  as  Officers  in  Charge  of  a  Navigational  Watch  (IMO,  1996C).  In  addition,  a  formal 
analysis  of  ARPA  operation  skill  and  knowledge  requirements  conducted  earlier  under  the 
direction  of  the  USCG  R&D  Center  provided  a  useful  functional  analysis  of  ARPA  operational 
requirements  (Sanquist,  Lee,  &  Rothblum,  1 994).  Finally,  basic  issues  and  concepts  were 
reviewed  in  selected  ARPA  instructional  texts  (Bole  &  Dineley,  1990;  Van  Wyck  &  Carpenter, 
1984). 

Following  this  initial  review  of  available  source  documents,  a  series  of  meetings  was  convened  to 
discuss  and  refine  the  set  of  assessment  objectives.  Meeting  attendees  included  experts  in 
navigation  and  ARPA  operation,  regulatory  requirements,  and  assessment  development.  In  these 
meetings,  lists  of  assessment  objectives  were  reviewed  and  refined  until  a  final  set  of  objectives 
was  specified.  In  developing  the  set  of  ARPA  assessment  objectives,  a  more  general  set  of 
functions  that  could  be  used  to  categorize  assessment  objectives  for  the  operation  of  a  range  of 
marine  electronic  equipment  emerged.  Six  general  categories,  representing  separate  aspects  of 
knowledge  and  skill  requirements  were  identified.  It  was  postulated  that  these  categories  would 
apply  to  the  assessment  of  other  electronic  equipment  operation,  such  as  Electronic  Chart  Display 
and  Information  Systems  (ECDIS)  and  large-scale  sensor  and  alarm  systems,  as  well  as  ARPA. 
These  six  general  categories  are: 

•  Equipment  initialization. 

•  Basic  understanding  of  equipment  output. 

•  Technical  limitations  of  the  equipment. 

•  Advanced  technical  operations. 

•  Broad  application  of  knowledge  and  skill  to  the  job. 
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•  Operational  warnings  and  system  tests. 

For  the  present  ARPA  application,  27  separate  assessment  objectives  were  organized  under  the 
six  assessment  objective  areas  presented  in  Table  4.  Appendix  A  provides  a  detailed  listing  of 
the  assessment  objectives  identified  in  the  present  ARPA  application,  along  with  the  citation 
from  the  STCW  Code  used  as  the  reference  in  developing  each  objective. 

Table  4.  ARPA  assessment  objective  areas. 

1 .  Setting  up  and  maintaining  displays. 

2.  Situation  assessment. 

3.  Knowledge  of  factors  affecting  ARPA  performance  and  accuracy. 

4.  Parallel  indexing. 

5.  Applications  of  COLREGS*  and  operation  of  trial  maneuvers. 

6.  Use  of  operational  warnings  and  system  tests. _ _ 

*  Collision  Regulations  (USCG,  1999) 

The  mariner  performance  specified  for  assessment  was  determined  on  the  basis  of  a  detailed 
review  of  each  assessment  objective,  the  conditions  necessary  for  eliciting  that  performance,  and 
a  preliminary  determination  of  the  specific  measures  of  performance  that  would  be  obtained  and 
assessed.  For  example,  one  of  the  ARPA  assessment  objectives  presented  in  Appendix  A  is  3.1  - 
Knowledge  of  effects  of  limitations  of  radar  range  and  bearing  on  the  accuracy  of  ARPA  data. 
One  such  limitation  is  that  when  two  or  more  vessels  have  different  ranges,  but  are  on  the  same 
bearing,  an  ARPA  system  will  often  take  longer  to  calculate  valid  target  data  than  if  the  targets 
are  on  different  bearings.  When  this  is  the  case,  a  mariner  must  wait  a  sufficient  period  of  time 
until  the  vessel  data  become  stable  and  valid. 

In  general,  application  of  the  proposed  method  to  ARPA  operation  resulted  in  a  comprehensive 
set  of  assessment  objectives.  The  development  of  these  objectives  required  substantial  resources 
and  time.  Even  in  this  trial  application,  for  which  the  STCW  Code  provides  detail  regarding 
mariner  proficiencies,  a  substantial  amount  of  analysis  was  required  to  define  a  complete  set  of 
assessment  objectives.  In  the  present  case,  it  was  found  that  the  presence  of  assessment 
development  experts  was  critical  in  defining  the  general  objectives  and  requirements  for 
assessment.  In  the  future,  the  need  for  having  assessment  developer  experts  present  throughout 
development  could  be  replaced  with  clear  guidance  that  can  be  applied  by  mariner  training 
experts. 

Assessment  Method 

The  STCW  Code  specifies  the  use  of  an  ARPA  simulator  as  the  preferred  method  for  assessing 
operator  proficiency.  In  the  present  application,  the  assessment  was  developed  using  the  high 
fidelity,  part-task  ARPA  simulator  at  the  USMMA  in  Kings  Point,  New  York.  At  the  Academy’s 
facility,  a  total  of  18  separate  operator  consoles  are  controlled  by  a  Norcontrol  simulator  system. 
Each  operator  console  has  an  operational  ARPA  console  that  displays  real-time  target  data  fed  by 
the  Norcontrol  system,  a  vessel  control  unit,  a  communications  unit,  and  a  chart  table  for 
reviewing  charts  and  plotting  traffic  situations. 
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It  is  important  to  note  that  our  results  here  should  not  be  interpreted  as  a  mandate  for  using  the 
specific  type  of  simulator  employed  in  this  study,  or  for  the  use  of  high-fidelity  simulators  in 
assessing  ARPA  operation.  Various  other  simulators  are  available  for  assessing  ARPA 
operation,  including  PC-based  simulators.  For  the  purposes  of  assessment,  the  capability  of  any 
simulator  to  support  assessment  and  training  can  be  evaluated  on  the  basis  of  the  degree  to  which 
the  specified  training  and  assessment  conditions  can  be  provided  by  the  simulator. 

For  21  of  the  27  ARPA  assessment  objectives,  it  was  determined  that  the  USMMA  simulator 
provided  the  best  assessment  method.  In  these  cases,  it  was  determined  that  real-time  ARPA 
operation  and  interpretation  would  provide  the  most  valid  method  of  assessment.  However,  for 
three  objectives  it  was  determined  that  basic  ARPA  knowledge  was  being  assessed.  For  these 
objectives,  it  was  decided  that  a  series  of  written  test  questions  would  serve  as  the  most  valid  and 
practical  method  of  assessment.  Additionally,  for  three  other  objectives  it  was  determined  that 
performance  was  highly  dependent  upon  the  type  of  equipment  being  operated.  In  these  cases, 
no  assessment  procedures  were  developed.  Appendix  A  indicates  the  selected  assessment 
method  for  each  of  the  present  ARPA  assessment  objectives. 

The  finding  regarding  the  use  of  written  questions  for  the  assessment  of  basic  knowledge  can 
probably  be  generalized  to  a  wide  range  of  mariner  proficiencies.  Most  maritime  jobs  require  the 
application  of  theoretical  or  procedural  knowledge.  Often,  one  can  expect  that  mariner  recall  or 
recognition  of  the  corresponding  information  is  a  prerequisite  to  successful  job  performance. 
When  a  written  or  oral  question  legitimately  reflects  the  requirements  of  the  job,  incorporating 
this  assessment  method  into  the  complete  assessment  procedure  should  increase  the  overall 
validity  and  reliability  of  the  assessment.  However,  it  is  also  important  to  recognize  that  written 
and  oral  questions  typically  represent  limited  aspects  of  job  conditions  and  requirements. 
Demonstrating  the  application  of  the  theoretical  and  procedural  knowledge  under  conditions  that 
reflect  normal  job  requirements  reflects  a  more  demanding,  realistic,  and  valid  method  of 
assessment. 

Assessment  Conditions 

In  the  current  application  of  the  proposed  assessment  development  method,  the  conditions 
required  for  each  assessment  objective  were  identified  and  grouped  into  six  separate  simulator 
exercises.  The  grouping  of  objectives  resulted  from  several  cycles  of  revision  and  refinement, 
with  the  basic  objective  of  establishing  a  realistic  series  of  events  that  represented  vessel  traffic 
conditions  that  could  be  expected  under  normal  operating  conditions. 

An  earlier  bridge  watchkeeping  assessment  development  effort  (Smith  et  al.,  1996)  demonstrated 
the  value  of  dividing  watchkeeping  exercises  into  two  segments.  In  the  first  segment,  the  vessel 
traffic  situation  evolves  without  allowing  the  candidate  to  maneuver  own  ship.  This  segment 
provides  a  basis  for  assessing  the  candidate’s  ability  to  initialize,  set-up,  and  operate  equipment, 
as  well  as  understand  and  interpret  equipment  displays.  In  the  second  segment,  the  candidate  is 
allowed  to  maneuver  the  vessel.  This  latter  segment  allows  for  confirmation  of  situation 
assessment  and  application  of  the  International  Rules  for  the  Prevention  of  Collisions  at  Sea 
(COLREGS)  (USCG,  1999).  This  basic  approach  was  also  found  to  be  useful  in  the  present 
application. 
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Each  of  the  six  ARPA  simulator  exercises  begins  with  own  ship  and  several  other  vessels 
displayed  on  the  screen  after  the  system  has  been  initialized  and  set  up.  Assessment  candidates 
are  given  specific  set-up  instructions  in  some  exercises  and  left  to  select  their  preferred  operating 
mode  in  others.  Specific  instructions  are  provided  regarding  the  intended  course  of  own  ship  and 
allowable  closest  point  of  approach  (CPA)  to  other  vessels.  Candidates  are  asked  to  record 
accurate  vessel  data  for  all  ships  while  standing  the  watch.  Candidates  are  also  given  control  of 
ship  course  and  speed  at  predetermined  times.  Exercises  vary  on  the  basis  of  traffic  conditions, 
system  alarms,  and  unexpected  system  malfunctions  that  occur. 

Figure  2  provides  an  example  of  the  situation  for  an  exercise,  showing  the  plot  of  all  vessels 
following  successful  navigation  during  exercise  C.  This  exercise  involves  an  open  waters 
situation  with  six  other  vessels  traveling  and  three  anchored  buoys.  During  this  exercise,  a  pre¬ 
programmed  own  ship  maneuver  occurs  automatically  prior  to  the  assessment  candidate 
assuming  control  of  the  vessel,  but  no  other  vessel  maneuvers  occur  until  own  ship  returns  to  its 
original  course.  This  exercise  was  designed  to  assess  seven  individual  objectives.  Appendix  B 
provides  assessment  exercise  general  descriptions,  assessment  objectives,  and  event  timelines. 


Figure  2.  Plot  of  an  example  simulator  exercise. 


18 


The  development  of  the  simulator  exercises  went  through  several  cycles  before  the  final  set  of  six 
exercises  was  completed.  During  this  process,  several  shorter,  more  discrete  exercises  that 
focused  on  individual  assessment  objectives  were  defined.  As  the  simulator  exercises  evolved, 
some  of  these  smaller  exercises  were  incorporated  into  longer  and  more  complex  exercises.  At 
the  completion  of  exercise  development,  a  basic  trade-off  between  the  complexity  of  the  exercise 
and  the  ease  of  performance  measurement  and  the  application  of  performance  standards  was 
identified.  Those  involved  in  this  application  of  the  assessment  development  method  noted  the 
inherent  advantage  of  less  complex  simulator  exercises  that  focus  on  a  limited  set  of  assessment 
objectives.  The  main  advantages  of  this  type  of  exercise  are  that  assessment  conditions  can  be 
more  tightly  controlled,  and  performance  standards  can  be  more  precisely  defined  and  applied. 

Proficiency  Criteria 

For  the  current  application,  performance  standards  were  established  through  consultation 
between  assessment  development  and  ARPA  experts.  Individual  assessment  objectives  were 
reviewed  and  performance  measures  were  defined  in  the  context  of  the  particular  simulator 
exercise  that  was  used.  Table  5  provides  an  example  of  the  performance  measures  and  standards 
corresponding  to  assessment  objective  2.6  -  The  speed  and  direction  of  target’s  relative 
movement,  and  the  identification  of  critical  echoes  (in  both  relative  and  true  motion  modes  of 
display.  For  this  assessment  objective,  there  are  four  separate  performance  measures  to  be 
obtained  for  each  vessel  up  to  three  times  during  the  course  of  an  exercise.  However,  a  single  set 
of  standards  can  be  applied  to  the  measures  throughout  the  exercises.  As  long  as  the  reported 
vessel  data  are  within  the  standards  presented  in  Table  5,  a  candidate  would  “pass”  this 
assessment  objective.  Appendix  B  provides  a  complete  list  of  measures  and  standards  for  the 
present  ARPA  application. 

Table  5.  Example  of  performance  measures  and  standards  for  assessment  objective 
2.6  -  Identification  of  critical  echoes:  Target  course,  speed,  CPA,  and  TCPA. 


Vessel  Data  Measures  and  Standards 

Course 

Speed 

CPA 

TCPA 

Standard  for  “Pass” 
Performance 

1+ 

o 

±  1  kn 

±  0.5  nm 

±  3  min 

Standard  for  “Fail” 
Performance 

Outside  performance  standards  for  “pass'" 

Although  the  current  application  yielded  a  comprehensive  set  of  individual  performance 
measures  and  standards,  a  more  integrated  set  of  proficiency  criteria  and  scoring  procedures  was 
not  developed  as  part  of  this  effort.  A  more  comprehensive  proficiency  criterion  is  needed  at  two 
levels:  first,  to  judge  if  performance  corresponding  to  an  individual  assessment  objective  with 
multiple  performance  measures  is  acceptable;  and  second,  to  judge  if  performance  across  the  full 
range  of  assessment  objectives  in  the  assessment  procedure  is  proficient.  At  the  time  of 
development,  the  lead  ARPA  expert  for  this  study  considered  the  development  of  such 
proficiency  criteria  as  premature.  A  comprehensive  pre-test  of  the  individual  measures  and 
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standards  with  a  representative  sample  of  academy  cadets  was  considered  necessary  prior  to 
establishing  proficiency  criteria. 

In  certain  cases,  the  ARPA  assessment  procedure  represents  a  unique  set  of  requirements. 
Preliminary  performance  standards  can  be  set  for  the  individual  measures  in  the  ARPA 
assessment.  However,  more  familiarity  with  how  candidates  perform  may  be  required  prior  to 
developing  proficiency  criteria  and  scoring  procedures.  In  general,  it  is  likely  that  complex 
watchstanding  job  requirements,  such  as  ARPA  operation,  will  require  substantial  trial 
application  before  setting  proficiency  criteria  and  scoring  procedures.  In  addition,  the  available 
guidance  regarding  development  of  more  integrated  scoring  procedures  was  very  limited  at  the 
time  of  trial  application.  Thus,  it  appears  that  additional  research  and  development  are  required 
to  provide  a  more  comprehensive  assessment  development  method  and  corresponding  practical 
guidance  that  addresses  establishing  proficiency  criteria  and  scoring  procedures. 


Assessment  Package 

In  the  present  ARPA  application,  development  of  assessment  materials  was  a  time-consuming 
process  that  required  several  iterations  of  review  and  refinement.  The  final  products  of  this  effort 
were  a  programmed  set  of  six  exercises  on  the  USMMA  ARPA  simulator  and  the  final 
assessment  materials.  For  the  most  part,  assessment  documentation  served  as  a  tool  to  track  the 
assessment  development  process  and  was  not,  in  itself,  an  undue  burden.  The  results  of  this 
documentation  effort  are  presented  in  the  appendices  to  this  report: 


•  Appendix  A: 

•  Appendix  B: 

•  Appendix  C: 

•  Appendix  D: 


ARPA  assessment  objectives,  conditions,  and  performance  measures. 

ARPA  assessment  simulator  exercise  specification  (including  performance 
measures  and  correct  operator  responses). 

Candidate  instructions  and  worksheets  for  ARPA  assessment  (including 
all  written  questions). 

Assessor  instructions  for  ARPA  assessment. 


The  following  documentation  is  not  provided  in  the  appendices  of  this  report;  however,  it  would 
be  required  for  a  complete  assessment  package: 


•  Reference  materials  for  candidate  review. 


•  Self-contained  candidate  pre-briefing  materials. 

•  Proficiency  scoring  procedures  and  record  forms. 


Summary  of  Application  Findings 

Overall,  this  trial  application  of  the  general  method  of  developing  assessment  procedures  found  it 
to  be  useful  in  guiding  the  development  of  an  objective  and  valid  set  of  performance-based 
assessment  procedures.  The  availability  of  researchers  expert  in  training  and  assessment  proved  a 
valuable  and  even  critical  resource  in  guiding  the  marine  educators  through  the  development 
process.  The  high-fidelity  ARPA  simulator  available  at  the  USMMA  was  an  excellent  testbed. 
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providing  the  capabilities  needed  to  support  the  assessment  of  this  very  complex  mariner 
proficiency.  During  the  trial,  several  aspects  of  the  assessment  method  that  required  further 
consideration  were  identified.  These  included  approaches  to  integrating  individual  performance 
measures  and  standards  into  proficiency  criteria  for  larger  units  of  performance.  Also  needed 
was  more  specific  guidance  on  the  useful  content  and  format  of  the  final  assessment  package  to 
be  handed  over  to  the  over-the-shoulder  assessor  and  the  candidate.  In  addition,  this  trial  had 
centered  on  the  use  of  a  simulator  in  an  academy  setting  and  little  consideration  had  been  given 
to  the  requirements  of  shipboard  assessment. 
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CONCLUSIONS  AND  RECOMMENDATIONS 


Conclusions 

This  report  describes  a  major  effort  to  examine  the  STCW  Code  and  the  USCG  requirements  for 
performance-based  assessments  and  to  ensure  that  the  best  practices  are  available  to  the  maritime 
industry  in  fulfilling  those  requirements.  The  effort  addressed  two  principal  objectives: 

•  Specify  and  refine  a  method  for  developing  performance-based  assessments  of  mariner  skills 
and  knowledge  that  is  responsive  to  IMO  and  USCG  guidance. 

•  Develop  a  performance-based  assessment  for  ARPA  operation  that  can  serve  as  one  example 
of  an  assessment  procedure  that  conforms  to  the  standards  of  competence  in  the  STCW  Code. 

To  fulfill  the  first  objective,  a  team  of  researchers  with  expertise  in  training  and  assessment 
worked  with  marine  educators  from  the  USMMA  to  prepare  a  step-by-step  method  of  developing 
assessments  that  was  especially  tailored  for  the  current  needs  of  the  industry.  To  test  and  refine 
the  method,  the  team  applied  it  to  the  assessment  of  proficiency  in  a  skill  emphasized  in  the 
STCW  Code,  the  operational  use  of  ARPA.  The  team’s  success  in  fulfilling  these  objectives  is 
documented  here.  Section  2  presents  their  systematic  five-step  method  for  the  development  of  an 
assessment  procedure;  Section  3  describes  their  experience  in  applying  it  to  ARPA.  The 
appendices  provide  their  ARPA  assessment  objectives,  the  needed  simulator  exercises, 
instructions  for  the  assessor  and  the  candidate,  and  worksheets  to  allow  others  to  recreate  their 
assessment  procedures. 

While  substantial  progress  was  made  on  these  ambitious  objectives,  the  trial  application 
identified  several  aspects  of  the  assessment  process  that  merited  additional  examination.  The 
team  had  available  the  USMMA  ARPA  laboratory,  a  high-fidelity,  real-equipment-based 
simulator,  and  did  not  need  to  concern  themselves  with  the  capabilities  of  the  setting  to  support 
their  assessments.  The  team  included  expertise  in  the  variety  of  disciplines  that  contribute  to 
assessment  development  and  had  dedicated  time  for  the  project.  Others  in  the  industry  are  not 
likely  to  have  these  ample  resources  available  and  will  need  guidance  on  developing  and 
conducting  assessments  with  fewer  resources.  In  addition,  the  trial  ARPA  assessment  identified 
limitations  to  the  guidance  provided  in  the  five-step  method.  While  the  method  was  effective  in 
guiding  the  development  of  performance  measures  and  standards  for  specific  mariner  actions,  it 
did  not  provide  scoring  techniques  for  integrating  these  measures  into  proficiency  criteria  for 
larger  units  of  mariner  performance.  Also,  the  team  found  that  further  guidance  was  needed  on 
the  content  of  the  instructions  to  be  provided  to  the  over-the-shoulder  assessor.  (Since  the 
method  is  limited  in  guiding  these  specific  aspects  of  assessment  development,  the  ARPA 
illustration  is  limited  there  as  well.  Future  users  of  these  ARPA  materials  should  consider 
augmenting  these  features.)  Later  phases  of  the  research  project  focused  on  all  these  additional 
needs.  The  resulting  project  reports  provide  the  additional  guidance  for  the  use  of  the  industry, 
as  described  below. 
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The  second  phase  of  the  research  project  provided  a  method  to  evaluate  simulators  in  their 
capability  to  support  the  mariner  in  realistically  demonstrating  the  performance  to  be  assessed, 
and  to  support  the  instructor/assessor  in  preparing,  controlling,  monitoring,  and  recording 
assessment  exercises.  The  general  simulator  evaluation  method  is  presented  in  a  second  project 
report,  along  with  example  exercises  and  worksheets  for  evaluating  PC-based  ARPA  simulators 
(Raby,  Forsythe,  McCallum,  &  Smith,  2000).  The  third  project  phase  focused  on  the  guidance 
that  maritime  instructors/assessors  would  need  to  apply  the  step-by-step  method  of  assessment 
development  without  the  assistance  of  assessment  experts.  The  researchers  worked  with 
maritime  instructors  to  prepare  and  refine  the  guidance  that  they  needed  for  developing  valid  and 
reliable  assessments.  Workshop  materials,  a  developer’s  manual,  and  sample  assessments  for 
both  deck  and  engine  room  proficiencies  are  provided  in  a  third  project  report  (McCallum, 
Forsythe,  Barnes,  Smith,  Macaulay,  Sandberg,  Murphy,  &  Jackson,  2000).  During  this 
additional  refinement  of  the  step-by-step  method,  techniques  for  combining  individual 
performance  measures  and  standards  into  proficiency  criteria  for  larger  units  of  performance 
were  perfected  and  are  included  in  the  developer’s  manual. 

The  fourth  and  final  phase  of  the  project  was  an  intense  examination  of  the  process  of  conducting 
over-the-shoulder  assessments,  especially  in  the  demanding  setting  of  a  commercial  ship.  The 
training  officers  of  a  major  shipping  company  worked  with  the  research  team  to  adapt  previously 
developed  assessments  to  the  operations  and  equipment  of  their  own  ships  and  to  prepare 
assessment  packages  for  use  by  their  regular  ship  officers  in  eonducting  trial  shipboard 
assessments.  A  discussion  of  what  was  learned  during  the  shipboard  trials  is  presented  in  the 
final  report  of  the  project  (McCallum,  Barnes,  Forsythe,  Smith,  Maynard,  Blanchard,  Hempstead, 
Martinez,  Murphy,  and  Jackson,  2000).  Also  included  are  an  assessor’s  manual  and  sample 
assessment  materials  for  deck  and  engine  proficiencies.  The  trials  provided  a  detailed 
understanding  of  what  is  needed  in  the  assessment  package  to  support  the  activities  of  the 
assessor  and  the  candidate. 

Recommendations  for  Implementation  of  the  Findings  and  Products 

For  the  most  effective  implementation  of  the  project’s  findings  on  performance-based  assessment 
and  its  supporting  products,  the  following  USCG  and  industry  actions  are  recommended. 

•  The  USCG  should  encourage  the  maritime  industry  to  use  the  performance-based  assessment 
method,  as  described  in  the  project  reports  and  materials.  As  a  first  step,  the  materials  in  this 
report  should  be  made  widely  available,  on  the  USCG  STCW  website 
(http://www.uscg.mil/stcw)  and  through  the  National  Technical  Information  Service  (NTIS). 
When  the  industry  becomes  more  familiar  with  the  method,  it  can  serve  as  the  basis  for  a  new 
NVIC.  In  addition,  the  USCG  should  submit  the  supporting  materials  to  the  IMO 
subcommittee  on  Standards  for  Training  and  Watchkeeping  and  recommend  their  further 
distribution. 

•  The  USCG  should  encourage  further  familiarity  with  the  performance-based  assessment 
approach  by  USCG  staff,  as  a  tool  for  discussions  of  assessment  with  the  industry  and  for  the 
review  of  procedures  submitted  for  approval. 
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•  The  USCG  should  encourage  the  review  and  use  of  the  performance-based  assessment 
method  and  materials  by  those  groups  who  are  dealing  with  the  important  technical  issues  of 
assessment.  These  groups  include  the  Maritime  Academy  Simulator  Committee  (MASC), 
the  Merchant  Marine  Personnel  Advisory  Committee  (MERPAC),  and  academy  committees 
appointed  by  the  Maritime  Administration  (MARAD)  to  address  STCW  issues.  The 
materials  can  provide  a  common  approach  and  a  common  basis  for  discussion. 

•  Those  in  the  industry  who  are  responsible  for  training  and  assessment  of  mariner  proficiency 
in  academies,  training  schools,  and  shipping  companies  should  make  use  of  the  performance- 
based  assessment  method  and  the  materials  presented  here  as  a  guide  for  their  own 
development  of  assessment  procedures.  All  the  project  materials  are  especially  appropriate 
for  inclusion  in  train-the-trainer  courses.  The  developer’s  manual  and  the  assessor’s  manual 
are  also  appropriate  for  those  who  do  not  take  such  courses. 
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GLOSSARY 


Assessor.  Anyone  who  conducts  an  assessment  or  evaluation  of  an  individual’s  proficiency. 
Although  the  use  of  the  term  assessor  is  retained  in  some  discussions  of  STCW  requirements, 
including  NVIC  4-97  on  company  roles  and  responsibilities,  the  term  designated  examiner  is 
used  for  assessor  in  the  U.S.  implementing  regulations. 

Assessment.  The  process  of  evaluating  whether  an  individual’s  performance  meets  established 
proficiency  criteria.  The  terminology  used  for  this  process  in  the  U.S.  implementing  regulations 
includes  both  an  examination  for  knowledge  and  an  assessment  based  on  a  practical 
demonstration  as  witnessed  by  a  designated  examiner. 

Assessment  conditions.  The  assessment  conditions  define  the  setting,  tools,  references,  aids, 
and  safety  precautions  that  are  in  place  at  the  time  that  a  candidate’s  proficiency  is  assessed. 

Assessment  objective.  A  critical  requirement  of  job  performance  that  can  be  measured  and 
assessed.  Assessment  objectives  include  skills  and  knowledge  required  by  the  job.  A  complete 
assessment  objective  description  includes  the  required  mariner  performance,  the  conditions  of 
assessment,  and  the  standards  of  performance  for  successful  accomplishment  of  the  objective. 

Assessment  procedures.  The  activities  that  are  conducted  in  administering  the  assessment  of  a 
candidate’s  proficiency.  The  term  assessment  procedure  can  be  used  to  describe  either  the 
actions  taken  or  the  written  instructions  and  activity  descriptions  that  are  carried  out  in 
conducting  an  assessment. 

Checklist.  A  type  of  subjective  measure  that  requires  the  assessor  to  observe  the  candidate’s 
performance,  or  outcome  of  performance,  and  choose  one  of  two  performance  levels,  typically 
labeled  “pass”  and  “fail.” 

Competence.  Broadly  defined  job  requirements  that  are  described  on  the  basis  of  training, 
experience,  and  job  skills.  The  standards  of  competence  as  defined  in  the  STCW  Code,  are  the 
“levels  of  proficiency  to  be  achieved  for  the  proper  performance  of  functions  on  board  ship.” 

Designated  examiner.  A  person  who  has  been  trained  or  instructed  in  techniques  of  training  or 
assessment  and  is  otherwise  qualified  to  administer  performance  assessment  procedures.  In 
practice,  the  designated  examiner  evaluates  whether  the  candidate’s  performance  meets  an 
established  proficiency  criterion  to  earn  credit  toward  the  license,  document,  or  endorsement. 
Further  details  on  the  qualifications  of  designated  examiner  can  be  found  in  NVIC  6-97. 

Duty.  An  ongoing  responsibility  within  a  job.  It  usually  requires  the  performance  of  multiple 
tasks  (e.g.,  officer  in  charge  of  the  engineering  watch,  lookout,  helmsman). 

Evaluation  criteria.  The  evaluation  criteria  comprise  the  general  standards  of  competence.  In 
practice,  the  evaluation  criteria  are  further  defined  on  the  basis  of  performance  measures, 
performance  standards,  and  proficiency  criteria.  This  term  is  defined  in  the  STCW  Code  as  the 
“entries  appearing  in  column  4  of  the  Specification  of  Minimum  Standard  of  Competence  tables 
in  part  A.” 
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Job.  A  post  of  employment  consisting  of  a  cluster  of  related  work  responsibilities  and  duties 
(e.g.,  chief  engineer,  third  mate,  able-bodied  seaman).  In  the  STCW  Code,  a  job  is  further 
defined  on  the  basis  of  licensure  level  (e.g.,  officer  in  charge  of  a  navigational  watch  on  ships  of 
500  gross  tonnage  or  more). 

Knowledge.  The  learned  concepts,  cues,  facts,  rules,  and  procedures  that  are  necessary  for 
proficient  performance  of  a  task  (e.g.,  knowledge  of  algebra,  knowledge  of  the  Navigation  Rules, 
knowledge  of  procedures  for  starting  the  main  engine). 

Objective  measure.  A  measure  that  relies  primarily  upon  measurement  apparatus  that  can  be 
calibrated  to  yield  highly  consistent  and  accurate  measurement  results. 

Performance  measure.  The  procedures  used  for  observing  and  recording  mariner  actions,  or  the 
outcome  of  those  actions.  Performance  measures  record  either  the  process  of  performance  or  the 
product  of  performance. 

Performance  standard.  The  acceptable  or  target  level  established  for  individual  performance 
measures.  Performance  measures  and  performance  standards  are  combined  on  the  basis  of 
scoring  procedures  to  establish  proficiency  criteria  for  a  proficiency. 

Proficiency.  An  individual’s  demonstrated  ability  to  meet  job  performance  requirements,  as 
established  on  the  basis  of  performance  measures,  performance  standards,  and  proficiency 
criteria. 

Proficiency  criteria.  The  scoring  procedures  applied  in  determining  the  proficiency  level  of  a 
candidate  on  the  basis  of  performance  measures  and  performance  standards. 

Qualified  instructor.  According  to  the  United  States  implementing  regulations,  “the  person 
who  has  been  trained  or  instructed  in  instructional  techniques  and  is  otherwise  qualified  to 
provide  required  training  to  candidates  for  licenses,  documents,  or  endorsements.”  Further 
details  on  the  qualifications  of  qualified  instructors  can  be  found  in  NVIC  6-97. 

Rating.  A  type  of  subjective  measure  that  requires  the  assessor  to  observe  the  candidate’s 
performance,  or  outcome  of  performance,  and  choose  one  of  three  or  more  performance  levels. 

Reliability.  The  consistency  of  a  measurement  procedure.  In  the  context  of  assessment, 
reliability  generally  can  be  defined  as  the  consistency  of  the  assessment  outcome  when  applied 
under  comparable  conditions.  The  reliability  of  an  assessment  establishes  the  maximum  level  of 
possible  assessment  validity.  That  is,  an  assessment  cannot  be  any  more  valid  than  it  is  reliable. 

Scoring  procedure.  The  defined  procedures  for  combining  individual  performance  measures 
and  performance  standards  in  the  application  of  proficiency  criteria. 

Skills  and  abilities.  The  behaviors  that  must  be  applied  in  successful  performance  (e.g.,  typing 
skills,  equipment  fault-finding  skills,  navigation  skills,  shiphandling  skills).  Measurable  and 
observable  skills  are  those  of  interest  in  proficiency  assessment. 

Subjective  measure.  A  measure  that  relies  primarily  upon  an  assessor’s  observation  and 
interpretation  of  mariner  performance  to  determine  the  assessment  outcome. 
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Task.  A  single,  observable  work  assignment  that  is  independent  of  other  actions  and  results  in  a 
valuable  outcome.  A  task  must  be  observable,  be  a  complete  work  assignment,  have  a  specific 
beginning  and  end,  and  be  able  to  be  measured  by  its  intended  product  or  outcome. 

Validity.  In  the  context  of  assessment,  validity  can  be  defined  as  the  degree  to  which  an 
assessment  accurately  reflects  the  skill,  knowledge,  and  performance  requirements  of  the  job. 
The  maximum  validity  of  an  assessment  is  established  on  the  basis  of  its  reliability.  That  is,  an 
assessment  cannot  be  any  more  valid  than  it  is  reliable. 
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APPENDIX  A— ARPA  Assessment  Objectives,  Conditions  and 
Performance  Measures 


Appendix  A  provides  a  detailed  description  of  the  27  ARPA  assessment  objectives  derived  from 
the  amended  STCW  requirements  (IMO,  1996).  The  appendix  is  divided  into  six  tables, 
corresponding  to  the  following  assessment  objectives: 

1.  Setting  up  and  maintaining  displays. 

2.  Situation  assessment. 

3.  Knowledge  of  factors  affecting  ARPA  performance  and  accuracy. 

4.  Parallel  indexing. 

5.  Application  of  the  International  Rules  for  the  Prevention  of  Collisions  at  Sea  (COLREGS) 
and  operation  of  trial  maneuver. 

6.  Use  of  operational  warnings  and  system  tests. 

Each  table  is  divided  into  four  columns.  In  the  first  column  each  assessment  objective  is 
described  in  detail.  Each  objective  was  derived  from  the  amended  STCW  Code  from  either  Table 
A-II/1  or  Section  B-I/12,  the  sections  in  which  ARPA  proficiency  standards  are  described.  The 
STCW  Code  reference  for  each  objective  is  provided  in  the  second  column  (IMO,  1996). 

In  the  third  column  the  simulated  conditions  necessary  to  measure  each  assessment  objective  are 
described,  and  the  simulation  exercises  developed  to  correspond  to  the  assessment  conditions  are 
cited.  For  example,  objective  1.6,  Display  characteristics  and  an  understanding  of  when  to  use 
ground-  or  sea-stabilized  modes,  requires  a  scenario  which  includes  navigation  down  a  narrow 
channel  with  a  cross  current  and  at  least  one  threat  vessel.  In  the  present  application,  the  exercise 
that  was  developed  to  meet  these  conditions  was  exercise  E. 

The  fourth  column  describes  the  performance  measures  for  each  test  objective.  These 
performance  measures  should  be  applicable  to  the  corresponding  assessment  objectives  as  long 
as  the  general,  requisite  assessment  conditions  are  met. 

Twenty-one  of  the  27  assessment  objectives  are  measured  using  simulator-based  exercises 
described  in  Appendix  B.  However,  three  objectives  are  measured  using  written  test  questions 
because  they  were  determined  to  be  the  most  logistically  feasible  assessment  method:  (1)  2.5  - 
Knowledge  and  recognition  of  historic  data  as  a  means  of  indicating  recent  maneuvering  of 
targets',  (2)  3.4  -  An  appreciation  of  the  IMO  performance  standards  for  ARPA,  in  particular  the 
standards  relating  to  accuracy;  and  (3)  6.3  -  Precautions  to  be  taken  after  a  malfunction  occurs. 
The  questions  for  each  of  these  objectives  follow  a  simulation  exercise  in  which  a  related 
objective  is  assessed. 

For  three  additional  objectives  it  was  determined  that  mariner  performance  depended  highly  on 
the  type  of  ARPA  equipment  used,  therefore  the  assessment  conditions  and  performance 
measures  for  these  objectives  are  to  be  determined  by  the  individual  assessor:  (1)  2.4  -  The  use 
of  graphic  representation  of  danger  areas',  (2)  6.2  -  Methods  of  testing  for  malfunctions  of 
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ARPA  systems  including  functional  self-testing',  and  (3)  6.4  -  Ability  to  perform  system  checks 
and  determine  data  accuracy  of  ARPA,  including  the  trial  maneuver  facility,  by  checking  against 
the  basic  radar  plot. 


Table  A-1.  ARPA  assessment  objectives  that  address  setting  up  and  maintaining  displays. 


Assessment  Objective 

STCW 

Reference 

Assessment  Conditions 

Performance  Measure 

1 . 1  The  selection  of  display 
presentation;  stabilized 
relative  motion  displays 
and  true  motion  displays 

32.2 

Sect.  B-l/12 

Exercise  A 

Initial  ARPA  set-up 

•  Instructions  for  set-up  of 
display  mode  and 
presentation 

1.1.1  Display  mode  and 
presentation 

1 .2  The  selection,  as 

appropriate,  of  required 
speed  and  compass  input 
to  ARPA 

32.4 

24.2 

24.1 

Sect.  B-l/12 

Exercise  A 

Initial  ARPA  set-up 

•  Instructions  for  set-up  of 
speed  and  compass  input 

1.2.1  Speed  log  input 

1 .2.2  Compass  input 

1 .3  The  selection  of  ARPA 
plotting  controls, 
manual/automatic 
acquisition,  vector/graphic 
display  of  data 

32.5 

Sect.  B-l/12 

Exercise  A 

Initial  ARPA  set-up 

•  Instructions  for  set-up  of 
acquisition  rings  and 
manual  target  acquisition 

1 .3.1  Acquisition  ring  (guard 
zone)  settings 

1.3.2  Targets  manually 
acquired 

1 A  The  selection  of  the  vector 
time  scale 

32.6 

Sect.  B-l/12 

Exercise  B 

Initial  ARPA  set-up 

•  Instructions  for  setting 
vector  time  scale 

1 .4.1  Vector  time  scale 

1 .5  The  use  of  exclusion 
areas  when  automatic 
acquisition  is  employed  by 
ARPA 

32.7 

Sect.  B-l/12 

Exercise  B 

Initial  ARPA  set-up 

•  Instructions  for  setting 
exclusion  area 

1.5.1  Exclusion  zone 
settings 

1 .6  Display  characteristics 
and  an  understanding  of 
when  to  use  ground-  or 
sea-stabilized  modes 

21.0 

Sect.  B-l/12 

Exercise  E 

Own  ship  navigating  down 
narrow  channel  with  cross 
wind/current  and  other  vessel 

1.6.1  Maintenance  of  ship 
position  and  course 

1 .6.2  Determination  of 
aspect  of  other  vessels 

Table  A-2.  ARPA  assessment  objectives  that  address  situation  assessment. 


Assessment  Objective 

STCW 

Reference 

Assessment  Conditions 

Performance  Measure 

2. 1  Understanding  the  criteria 
for  the  selection  of  targets 
by  automatic  acquisition 

25.1 

Sect.  B-l/12 

Exercises  A  and  B 

ARPA  set-up 

•  Instructions  for  setting  of 
guard  zones 

2.1.1  Guard  zone  settings 
(same  as  performance 
measure  1.3.1) 

2.2  Appreciation  of  the  uses, 
benefits  and  limitations  of 
ARPA  operational 
warnings 

27.0 

Sect.  B-l/12 

Exercise  B 

ARPA  set-up 

•  Instructions  for  setting  safe 
limit 

Safe  limit  settings 

Safe  limit  warning 

Guard  zone  warning 
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Table  A-2.  ARPA  assessment  objectives  that  address  situation  assessment.  (Continued) 


Assessment  Objective 

STCW 

Reference 

Assessment  Conditions 

Performance  Measure 

2.3  Detection  and 

identification  of  false 
echoes,  sea  return, 
racons,  and  Search  and 
Rescue  Transponders 
(SARTs) 

Table  A 

Exercise  B 

Racon  included  in  exercise 

SARTs  are  not  currently 
available  at  USMMA  ARPA 
simulator 

2.3.1  Identification  of  racon 
code 

2.4  The  use  of  graphic 

representation  of  danger 
areas 

30.1 

Sect.  B-l/12 

Due  to  variations  in  equipment 
characteristics,  assessor- 
determined 

Assessor-determined 

2.5  Knowledge  and 

recognition  of  historic  data 
as  a  means  of  indicating 
recent  maneuvering  of 
targets 

31.0 

Sect.  B-l/12 

Exercise  A 

Instruction  to  demonstrate 
vessel  history  display 

Written  questions: 

Display  of  vessel  history 
patterns  (administered  at  the 
end  of  exercise  A) 

2.5.1  Demonstration  of 
vessel  history  display 

2.5.2  Identification  of  vessel 
history  patterns  on 
paper  facsimile  of  an 
ARPA  display 

2.6  The  speed  and  direction 
of  target’s  relative 
movement,  and  the 
identification  of  critical 
echoes  (in  both  relative 
and  true  motion  modes  of 
display) 

26.0,  34.1, 

34.2,  34.3, 

34.4 

Sect.  B-l/12 

Exercise  A 

Dead  in  the  water  target 

Exercise  C 

True  motion 

Exercise  D 

Relative  motion 

Instruction  to  report  accurate 
speed  and  direction  of  target 
relative  movement  as  soon  as 
possible 

2.6.1  Time  of  reporting 
speed  and  direction  of 
target  relative 
movement 

2.6.2  Reported  speed  and 
direction  of  target 
relative  movement 

2.6.3  Reported  speed  and 
direction  of  dead  in  the 
water  (DIW)  target 

2.7  Detecting  course  and 
speed  changes  of  targets 
and  the  limitations  of  such 
information  (in  both 
relative  and  true  motion 
modes  of  display) 

26.0 

34.5 

Sect.  B-l/12 

Exercise  C 

True  motion 

Exercise  A  and  Exercise  D 

Relative  motion 

Other  vessel  course  and 
speed  changes 

Instruction  to  update  vessel 
data  as  required 

2.7.1  Time  of  reporting  other 
vessel  data  following 
other  vessel  course/ 
speed  change 

2.7.2  Reported  vessel  data 
following  course/speed 
change 

2.8  The  effect  of  changes  In 
own  ship’s  course  or 
speed  or  both  (in  both 
relative  and  true  motion 
modes  of  display) 

26.0 

34.6 

Sect.  B-l/12 

Exercise  C 

True  motion 

Exercise  A 

Relative  motion 

Own  vessel  course  or  speed 
change 

Instruction  to  update  vessel 
data  as  required 

6.3.1  Time  of  reporting  other 
vessel  data  following 
own  ship  course/speed 
change 

2.8.2  Reported  vessel  data 
following  own  ship 
course/speed  change 
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Table  A-3.  ARPA  assessment  objectives  that  address  knowledge  of  factors  affecting 

performance  and  accuracy. 


Assessment  Objective 

STCW 

Reference 

Assessment  Conditions 

Performance  Measure 

3.1  Knowledge  of  effects  of 
limitations  of  radar  range 
and  bearing  on  the 
accuracy  of  ARPA  data 

24.1 

Sect.  B-l/12 

Exercise  F 

Three  vessels  on  same  initial 
bearing,  with  two  on  collision 
course  and  fourth  vessel  on  a 
different  bearing 

Instructions  to  report  valid  target 
data  as  soon  as  possible 

3.1.1  Reported  vessel  data 

3.2  The  circumstances  causing 
Target  swap”  and  their 
effects  on  display  data 

25.4 

Sect.  B-l/12 

Exercise  C 

Vessels  in  open  water  passing 
one  another  and  buoys 

3.2. 1  Identification  of  pairs 
of  targets  where  swap 
is  likely  to  occur 

3.3  The  effects  on  tracking  of 
Tosf  targets  and  target 
fading 

25.3 

Sect.  B-l/12 

Exercise  A 

Loss  of  target  track  and 
sounding  of  Target  lost”  alarm 

6.3.1  Identification  of  lost 
target  alarm 

3.3.2  Reacquisition  of  lost 
target 

3.4  An  appreciation  of  the  IMO 
performance  standards  for 
ARPA,  In  particular  the 
standards  relating  to 
accuracy 

22.0 

Sect.  B-l/12 

Written  questions 

(administered  at  the  end  of 
exercise  F) 

3.4.1  Correct  responses  to 
written  questions 

Table  A-4.  ARPA  assessment  objectives  that  address  parallel  indexing. 


Assessment  Objective 

STCW 

Reference 

Assessment  Conditions 

Performance  Measure 

4.1  Plotting  parallel  index  lines 
to  maintain  position  on 
planned  course 

Use  of  Radar 
and  ARPA 

Table  A-ll/1 

Exercise  B 

Instruction  to  draw  parallel  index 
lines  to  maintain  three  miles  from 
land 

4.1.1  Location  and  bearing 
of  parallel  index  lines 

4.2  Using  parallel  index  lines  to 
identify  time  of  maneuver 

Use  of  Radar 
and  ARPA 

Table  A-ll/1 

Exercise  B 

Instruction  to  change  own  vessel 
course  in  accordance  with 
parallel  index  lines 

6.3.1  Ship  heading  at 
predetermined  time  in 
exercise 

4.2.2  Ship  distance  from 
parallel  index  lines  at 
predetermined  time  In 
exercise 
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Table  A-5.  ARPA  assessment  objectives  that  address  application  of  COLREGS  and 

operation  of  trial  maneuver. 


Assessment  Objective 

STCW 

Reference 

Assessment  Conditions 

Performance  Measure 

5. 1  The  benefit  of  switching 
between  true  and  relative 
vectors 

30.4 

Sect.  B-i/12 

Exercise  D 

Relatively  complex  and  varied 
traffic  conditions,  including  other 
vessel  maneuvers 

Request  to  identify  lights  that 
would  be  seen  for  each  vessel 

5.1.1  Identification  of  other 
vessels*  lights  based 
on  ARPA  display 

5.2  Analysis  of  potential 
collision  situations  from 
displayed  Information, 
determination  and 
execution  of  action  to  avoid 
close-quarters  situations  in 
accordance  with 

COLREGS 

35.0 

Sect.  B-I/12 

Exercise  D 

Relatively  complex  and  varied 
traffic  conditions,  Including  other 
vessel  maneuvers 

Request  to  Identify  whether  or 
not  there  is  a  risk  of  collision  with 
each  of  the  other  vessels 

Request  to  identify  applicable 
COLREGS  for  each  of  the  other 
vessels 

6.3.1  Identification  of 
whether  or  not  there 
is  a  risk  of  collision 
with  other  vessels 

5.2.2  Identification  of 
applicable  COLREGS 
for  other  vessels 

5.3  The  operation  of  the  trial 
maneuver  facility 

34.7 

Sect.  B-I/12 

Exercise  C  (true  motion) 

Exercise  D  (relative  motion) 

Relatively  complex  and  varied 
traffic  conditions,  including  other 
vessel  maneuvers 

Instruction  (at  specific  time)  to 
calculate  required  new  course 

when  vessel  bearing _ is  at 

_ distance,  to  maintain  a 

minimum  CPA  of _ . 

6.3.1  Calculated  new 

course 

6.3.1  Maintenance  of 
minimum  CPA 
between  own  ship 
and  all  other  vessels 

5.3.3  Resumption  of  base 
course  after 
completion  of  trial 
maneuver 

Table  A-6.  ARPA  assessment  objectives  that  address  use  of  operational  warnings  and 

system  tests. 


Assessment  Objective 

STCW 

Reference 

Assessment  Conditions 

Performance  Measure 

6.1  Performance  checks  of 
radar,  compass,  speed 
input  sensors  and  ARPA 

32.8 

Sect.  B-I/12 

Exercise  C 

Speed  log  error  during  open 
waters  operations 

6.1.1  Identification  of  and 
response  to  speed 
log  error 

6.2  Methods  of  testing  for 
malfunctions  of  ARPA 
systems  including 
functional  self-testing 

28.1 

Sect.  B-I/12 

Assessor-determined 

Assessor-determined 

6.3  Precautions  to  be  taken 
after  a  malfunction  occurs 

28.2,  30.3 

Sect.  B-I/12 

Written  questions 

Questions  concerning  the  validity 
of  specific  types  of  vessel  data, 
given  certain  types  of 
malfunction 

(Administered  at  the  end  of 
exercise  C) 

6.3.1  Recognition  of  which 
display  data  are  valid 
after  an  equipment 
malfunction  occurs 
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Table  A-6.  ARPA  assessment  objectives  that  address  use  of  operational  warnings  and 

system  tests.  (Continued) 


Assessment  Objective 

STCW 

Reference 

Assessment  Conditions 

Performance  Measure 

6.4  Ability  to  perform  system 
checks  and  determine  data 
accuracy  of  ARPA, 
including  the  trial 
maneuver  facility,  by 
checking  against  basic 
radar  plot 

33.0 

Sect.  B-l/12 

Assessor-determined 

Assessor-determined 

A-6 


APPENDIX  B — ARPA  Assessment  Simulator  Exercise  Specification 


Appendix  B  describes  in  detail  the  six  ARPA  simulation  exercises  developed  using  the  present 

methodology  to  measure  21  of  the  objectives  detailed  in  Appendix  A. 

Exercises  A,  B,  C,  D,  E  and  F  each  are  divided  into  five  sections: 

.  General  Description,  including  type  of  waters,  weather  considerations,  complexity  of  the 
traffic  situation,  and  details  about  which  targets  maneuver. 

•  Assessment  Objectives,  including  the  number  and  description  of  each  test  objective. 

.  Vessel  Data  at  the  Beginning  of  the  Exercise,  including  own  ship  initial  course  and  speed, 
and  initial  bearing,  range,  course  speed,  CPA,  and  TCPA  for  all  targets. 

•  Detailed  Description  of  the  events  that  occur  during  the  exercise,  including  the  time  and 
phase  when  they  occur,  as  well  as  the  assessment  objective  and  performance  measured  in 
each  event,  and  the  correct  operator  response  (the  performance  standard)  for  each 
performance  measure. 

•  Mechanical  Plot  of  the  Scenario,  with  a  plan  view  graphical  depiction  of  the  ships,  buoys, 
racons,  and  land  present  in  each  exercise. 


B-1 


Q> 

.J2 

Si 

s 

uj 


CO 

D 


a 

<u 


<D 

00 

oo 

(D 

> 

(D 


C 

o 


o 

(0 

0 

Q 

75 

w 

0 

C 

0 

o 


o 

;-t 

CD 

X 

> 

a 

G 

<o 

X 

4—4 

u 

a 

2 

w 

*iC 

D 

G 

DO 

G 

{-t 

D 

G 

O 

4—4 

oo 

G 

oo 

G 

Ui 

D 

G 

G 

04 

X 

oo 

G 

D 

CL 

G 

O 

o 

• 

• 

(U 

a> 

CU 

oo 

oo 

<U 

O 

D 

'O 

<o 


a 

.2 

:3 

a 

13 
■<— > 
(D 
OD 

Ui 

c3 


G 


t)i) 

G 


G 

oo 

2 


(U 

tuo 

Ui 

G 

toJO 

G 

*> 

O 

S 

o 


oo 

G 


<D 

G 


00 

o 


^  Q  U 


CD  <U 
W)  DO 

H  H 


0 

0 

_> 

’>*3 

O 

0 

‘JBT 

O 

c 

0 

E 

0 

0 

0 

0 

< 


G 

O 

’_r^ 

G 

■4— » 

G 

a> 

oo 

a 

& 

■E. 

oo 


O 

G 


o 

GO 


G 

4— » 

G 

'G 


>> 

G 

oo 

•  r— H 

"G 

O 

1g 

a. 

G 

Ui 

DO 

O 

o 

G 

> 

^  O 
.S  G 


oo 

oo 

G 

CL, 

a 

o 

o 

G 

G 

G 

G 

(D 

(D 

CL 

oo 

L-t 

O 

CL 


oo 

'o 

U4 

-i— > 

G 

O 

O 

DO 

G 


CL, 


G 

O 


O 

(L> 


O  <D  <D 
^  CO  in 


<N  CO 


u* 

& 

-a 

G 

G 

G 

O 

G 

Ui 
4— > 

OO 

G 

O 

a 

(D 

G 

< 

CU 

Pi 

< 

DO 

G 

a> 

> 

G 

G 

G 

G 

a 

G 

G 

O 

a 

DO 

G 

G 

O 

G 

G 


00 

G 

G 

a 

G 

oo 

G 

G 

4— > 

G 

G 

O 


(U 

G 

O 

SI 

G 

G 

DO 


G 

O 


O  *;r! 


CL 

G 


00 

(D 

O 

JP 

O 

(D 

'g 

.2 

*4— > 

’C 

O 


G 

O 


^  Uh 

2  Q 


G 

DO 

O 

O 

(U 

Pi 


<D 

G 

G 

G 

G 

(D 

Q 


UO  MD 

(si  <N 


G 

.2 

*4-> 

o 

a 

(D 

> 

G 

a 


G 

O 

►  »-H 

G 

a 


XI 

o 

G 

oo 


2  o 


G  G 


G 

G 

G 

oo 

G 

O 
•  ^ 

•4— > 

G 

> 

4— > 

G 

O 

G 

G 

CD 

Ui 

<D 

oo 

Ut 

G 

O 

O 

13 

oo 

oo 

<o 

> 


G 

2 

*4—4 

o 

a 

(D 

> 

W 

2 

2 


00 

•2  c 


X! 

4^ 

G 

G 

G 


(U 

DO 

G 


00 

(D 

DO 

G 

G 

X 

o 

G 

<D 

(D 

CL 

c/3 

G 

G 

G 

(D 

CO 

U4 

G 

O 

o 

DO 

.a 

*4—4 

O 

(D 

a> 

Q 


X 

o 

X 

^-1 

o 

G 

<D 

<D 

CL 

oo 

u, 

O 

CD 

oo 

G 

O 

O 


G 

& 

O 

.a 

00 

CD 

DO 

G 

G 

X 

O 


o 

L— t 

CD 

CD 

X 

H 


D 

DO 


L-i 

o 

DO 

G 
•  ^ 

u, 

4—4 

G 

O 

4-4 

o 

pq 


r-  oq  CO 
(N  r4  CO 


G 


X 

2 

a 

CL 

w 

o 

o 

o 

o 


G 

G 

G 

u 

CO 

CO 

0^ 


a> 

CO 

CD 

U 

O) 

X 

U 


CQ 

3 

G 

H 


< 

Q. 

O 

H 

not  applicable 

o 

o 

CG 

cp 

— 

o 

o 

o 

o 

o 

o 

o 

o 

o 

•G 

o 

o 

q 

q 

o 

< 

Q. 

O 

(D 

X 

G 

.2 

Q. 

CL 

G 

O 

C 

o 

CO 

o 

d 

G; 

1“ 

o 

cvi 

CO 

CM 

G 

0) 

0) 

Q. 

0 

o 

<M 

5 

Q 

CO 

CM 

lO 

CM 

CO 

T~ 

(D 

CO 

3 

O 

o 

O 

o> 

O 

5 

Q 

o 

o> 

o 

o 

o 

o 

o 

CJ> 

o 

o 

o 

CM 

0) 

O) 

c 

(0 

cr 

G 

X 

G 

O 

■q. 

CL 

G 

O 

c 

G" 

d 

o 

d 

(O 

d 

CM 

CO 

o 

D) 

c 

*k. 

(0 

o 

CD 

G 

X 

G 

o 

“q. 

Q. 

G 

O 

cr 

CO 

r- 

o 

o 

o> 

o 

CO 

lO 

00 

CM 

o 

in 

o 

I) 

CO 

CO 

O 

> 

CL 

X 

0 

c 

O 

CQ 

o 

Q 

Uj 

B-2 


B-3 


For  targets  on  parallel  course  and  same  speed,  TCPA  is  not  applicable 


HS-90.  NORCONTROL  Norway 


Figure  B-1.  Plot  of  ARPA  assessment  exercise  A. 
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Table  B-4.  Detailed  description  of  exercise  B. 
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Table  B-4.  Detailed  description  of  exercise  B.  (Continued) 
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Figure  B-2.  Plot  of  ARPA  assessment  exercise  B. 
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Table  B-8.  Detailed  description  of  exercise  D. 
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Figure  B-5.  Plot  of  ARPA  assessment  exercise  E, 
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Table  B-12.  Detailed  description  of  exercise  F. 
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PLOT  INTEnVAL  90  SEC 
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Figure  B-6.  Plot  of  ARPA  assessment  exercise  F. 


APPENDIX  C — Candidate  instructions  and  Worksheets  for  ARPA 
Assessment 


Appendix  C  provides  the  vessel  cards,  instructions  and  worksheets  for  the  candidates  whose 
ARPA  proficiency  is  assessed  using  the  present  exercises.  There  are  two  vessel  cards:  one  for 
exercise  E  and  another  for  exercises  A,  B,  C,  D  and  F.  Exercises  A,  B,  C,  D,  E,  and  F  each  have 
a  candidate  worksheet  with  instructions  that  correspond  to  exercise  events  involving  the 
candidate.  Instructions  include  the  exercise  time  of  day,  the  settings  for  own  ship  course  and 
speed,  and  the  settings  for  other  pertinent  display  information,  such  as  safe  limits  CPA  and 
TCPA,  guard  zone,  exclusion  area,  and  vector  time  scale.  The  candidate  worksheets  also  include 
tables  in  which  vessel  data  can  be  recorded  at  various  times  during  an  exercise.  Instructions  for 
own  ship  maneuvers,  parallel  indexing  and  other  candidate  actions  are  also  included  on  these 
worksheets. 

Candidates  respond  to  simulator-based  questions  directly  on  the  worksheet  for  each  exercise.  For 
exercises  A,  C,  and  F  a  second,  separate  worksheet  with  multiple-choice,  true/false  or  short 
answer  test  questions  was  designed.  After  the  simulations  end  in  exercises  A,  C,  and  F,  the 
assessor  collects  the  worksheets  for  the  simulations  and  administers  the  test  questions  on  the 
second  worksheet. 
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e  C-1.  Vessel  card  for  exercises  A,  B,  C,  D,  and  F. 
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Figure  C-2.  Vessel  card  for  exercise  E. 


ARP  A  Assessment  Exercise  A 


Name _ _ _ —  ARPA  Unit _ 

Exercise  Time:  0100:00  Own  Ship  Course:  090°  Own  Ship  Speed:  20.0  knots 
Section  A 

1 .  Initialize  ARPA  to  North-up,  stabilized,  relative  motion  mode  and  be  prepared  to 
demonstrate  all  types  of  display  presentations  and  modes  available  on  your  ARPA  unit. 

2.  Set  guard  ring  (Raypath)  to  6  nm. 

3.  Set  guard  rings  (Raycas  V)  to  5  nm  &  9  nm. 

4.  Set  exclusion  line(s)  3  nm  north  of  you. 

5.  Set  automatic  acquisition  rings. 

6.  As  soon  as  your  initial  ARPA  settings  have  been  checked  by  the  instructor,  turn  off  automatic 
acquisition. 

7.  Manually  acquire  all  targets  on  your  display. 

8.  As  soon  as  you  are  underway,  record  vessel  data,  and  include  the  time  you  recorded  the  data 


for  each  target  (e.g.  0105:00). 


initial  Bearing 

Target 

Course 

Speed 

CPA 

TCPA 

Time 

073 

A 

090 

B 

131 

C 

285 

D 

050 

E 

9.  Your  ship  is  in  a  pre-programmed  mode,  and  it  changes  course  during  the  exercise.  After  the 
own  ship  course  change,  record  new  vessel  data,  and  include  the  time  you  recorded  the  data 
for  each  target  (e.g.  0101:00): 


Initial  Bearing 

Target 

Course 

Speed 

CPA 

TCPA 

Time 

073 

A 

090 

B 

131 

C 

285 

D 

050 

E 

10.  When  asked  by  the  assessor,  demonstrate  the  target  history  capability  of  your  ARPA  unit. 
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11.  Identify  any  vessels  that  change  course  or  speed  and  record  new  vessel  data,  and  include  the 
time  you  recorded  the  data  for  each  target: 


Initial  Bearing 

Target 

Course 

Speed 

CPA 

TCPA 

Time 

073 

A 

090 

B 

131 

C 

285 

D 

050 

E 
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ARP  A  Assessment  Exercise  A 


Name _ ARPA  Unit _ _ 

Section  B 

12.  Using  the  attached  facsimile  of  an  ARPA  display,  respond  to  the  following  questions 
regarding  vessel  history  trails.  Assume  true  vectors  are  displayed  with  vector  length  as 
12  minutes  and  a  12-mile  range  scale,  and  two-minute  history  intervals. 

a.  What  target,  if  any,  changed  course  in  the  last  two  minutes? _ 

b.  What  target(s),  if  any,  changed  course  several  minutes  ago? _ 

c.  What  target,  if  any,  is  increasing  speed?  _ 

d.  What  target  is  either  dead  in  the  water  or  recently  acquired? _ 

e.  What  target  has  been  tracked  for  at  least  three  minutes 

but  less  than  seven  minutes?  _ 
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ARP  A  Assessment  Exercise  B 


Name _ _ ARPA  Unit - 

Exercise  Time:  1000:00  Own  Ship  Course:  255°  Own  Ship  Speed:  20  knots 

1 .  Initialize  your  ARPA  unit  with  the  display  presentation  and  mode  of  your  choice. 

2.  Set  safe  limits  to  2  nm  CPA  and  24  minutes  TCPA. 

3.  Set  vector  time  scale  to  6  minutes. 

4.  Set  guard  zone  to  3  nm. 

5.  Draw  parallel  index  lines  to  maintain  3-mile  distance  from  land  north  of  you. 

6.  Set  exclusion  line(s)  2.5nm  south  of  own  ship. 

7.  As  soon  as  you  are  underway,  identify  the  code(s)  for  any  racon(s)  present: 

8.  Manually  acquire  all  pertinent  targets  on  your  display. 

9.  Record  vessel  data,  and  include  the  time  you  recorded  the  data  for  each  target  (e.g.  1005:00): 


Initial  Bearing 

Target 

Course 

Speed 

CPA 

TCPA 

Time 

230 

A 

255  @  11.7  nm 

B 

255  @  6.0  nm 

C 

069 

E 

296  @  3.0  nm 

K 

10.  Record  warnings  that  sound,  if  any,  and  the  time  they  occurred: 


Warning 

Time 

1 1.  If  one  or  more  warnings  sound,  take  the  appropriate  action  for  each  warning. 

12.  At  approximately  1018:00,  maneuver  to  course  270°  keeping  the  point  of  land  at  3nm,  in 
accordance  with  parallel  index  lines. 
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13.  Identify  any  vessels  that  change  course  or  speed  and  record  new  vessel  data,  and  include  the 
time  you  recorded  the  data  for  each  target: 


Initial  Bearing 

Target 

Course 

Speed 

CPA 

TCPA 

Time 

230 

A 

255  @  11.7  nm 

B 

255  @  6.0  nm 

C 

069 

E 

296  @  3.0  nm 

K 
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ARP  A  Assessment  Exercise  C 

Name _ ARPA  Unit - 

Exercise  Time:  0800:00  Own  Ship  Course:  180°  Own  Ship  Speed:  15.0  knots 

Section  A 

1 .  Initialize  ARPA  in  true  motion  mode. 

2.  Set  CPA  safe  limit  to  3  nm  and  TCPA  safe  limit  to  20  minutes. 

3.  As  soon  as  you  are  underway,  initialize  speed  log. 

4.  Acquire  all  targets  except  three  buoys  bearing  125°,  130°,  132°  true. 

5.  Record  data  for  all  targets,  and  include  the  time  you  recorded  the  data  for  each  target  (e.g., 
0810:00): 


initial  Bearing 

Target 

Course 

Speed 

CPA 

TCPA 

Time 

090° 

A 

158° 

B 

210° 

C 

270° 

D 

304° 

E 

005° 

F 

090° 

G 

6.  At  time  08:04,  identify  all  targets,  if  any,  that  have  target  swap  potential: 


7.  Calculate  and  execute  course  change  at  or  before  0808:00  for  a  min.  CPA  of  3  nm. 

Calculated  course  change: _ .  Resume  base  course  as  soon  as  possible. 

8.  After  executing  the  course  change,  record  accurate  vessel  data,  and  include  the  time  you 
recorded  the  data  for  each  target: 


Initial  Bearing 

Target 

Course 

Speed 

CPA 

TCPA 

Time 

090° 

A 

158° 

B 

210° 

C 

270° 

D 

304° 

E 

005° 

F 

090° 

G 
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9.  Do  not  adjust  your  vessel’s  course  or  speed  between  0812:00  and  0818:00;  however,  return 
to  base  course  as  soon  as  possible  after  0818:00,  and  maintain  at  least  a  minimum  CPA  of 
3nm. 

10.  Record  warnings  that  sound,  if  any,  and  the  time  they  occurred: 


Warning 

Time  of  occurrence 

1 1.  If  one  or  more  warnings  sound,  take  the  appropriate  action  for  each  warning. 

12.  Identify  target(s),  if  any,  that  changed  course  or  speed  during  the  exercise,  and  include  the 
time  you  recorded  the  data  for  each  target: 


Initial  Bearing 

Target 

Course 

Speed 

CPA 

TCPA 

Time 

090° 

A 

158° 

B 

210° 

C 

270° 

D 

304° 

E 

005° 

F 

090° 

G 
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ARPA  Assessment  Exercise  C 


Name 


ARPA  Unit 


Section  B 

13.  After  the  simulation,  respond  to  the  following  questions  regarding  the  accuracy  of  ARPA 
information  during  equipment  malfunction.  Assume  that  the  vessel  data  in  question  are; 
(A)  bearing,  (B)  CPA,  (C)  TCPA,  (D)  range,  (E)  course,  (F)  speed  or  (G)  none  of  these. 
Your  response  may  include  more  than  one  type  of  data. 

a.  Assume  the  compass/heading  input  is  incorrect.  Which,  if  any,  of  the  above  vessel  data 
would  be  correct? 


b.  Assume  the  speed  log  input  is  incorrect.  Which,  if  any,  of  the  above  vessel  data  would  be 
correct? 


c.  Assume  you’re  relying  solely  on  ARPA,  and  the  ARPA  target  tracking  capability  fails. 
What,  if  any,  automatic  ARPA  information  is  still  available? 
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ARP  A  Assessment  Exercise  D 

Name _ _ _ _ _ ARPA  Unit _ 

Exercise  Time:  0600:00  Own  Ship  Course:  180°  Own  Ship  Speed:  15.0  knots 

1.  Initialize  ARPA  in  relative  motion  mode. 

2.  Acquire  all  targets. 

3.  Assess  the  COLREGS  situation  for  each  target: 


Initial  Bearing 

Target 

Risk  of  Collision? 
(Y/N) 

COLREGS 

090 

A 

158 

B 

210 

C 

270 

D 

304 

E 

040 

F 

180 

G 

4.  Record  vessel  data  before  own  ship  course  change,  and  include  the  time  you  recorded  the 
data  for  each  target: 


Initial  Bearing 

Target 

Course 

Speed 

CPA 

TCPA 

Time 

090 

A 

158 

B 

210 

C 

270 

D 

304 

E 

040 

F 

180 

G 

5.  Record  what  lights  are  seen  on  each  vessel  on  or  before  06: 10:00: 


Initial  Bearing 

Target 

Lights 

090 

A 

158 

B 

210 

C 

270 

D 

304 

E 

040 

F 

180 

G 
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6.  Identify  any  vessels  that  change  course  or  speed  and  record  new  vessel  data,  and  include  the 
time  you  recorded  the  data  for  each  target: 


Initial  Bearing 

Target 

Course 

Speed 

CPA 

TCPA 

Time 

090 

A 

158 

B 

210 

C 

270 

D 

304 

E 

040 

F 

180 

G 

7.  Calculate  and  execute  course  change  when  target  C  is  at  6  nm  for  at  least  a  3  nm  CPA. 

Calculated  course  change: _ 

8.  Does  the  action  to  avoid  target  C  bring  another  vessel  into  an  unacceptable  situation?  If  so, 
what  will  you  do? 


9.  Resume  base  course  as  soon  as  possible  while  maintaining  at  least  the  minimum  CPA  on  all 
targets.  If  this  is  not  possible,  indicate  “impossible”  below: 
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ARPA  Assessment  Exercise  E 


Name _ ARPA  Unit _ 

Exercise  Time:  1100:00  Own  Ship  Course:  180°  Own  Speed:  15.7  knots 
No  VHP  radio  or  pilot  is  available  during  this  exercise;  a  chartlet  (figure  C-4)  is  attached. 


1.  Initialize  your  ARPA  unit  with  a  ground-stabilized  presentation  mode. 

2.  As  soon  as  you  are  underway,  set  autodrift  (groundlock)  to  target  bearing  172°  at  4  nm,  and 
manually  acquire  any  northbound  target(s). 

3.  Avoid  collisions  or  groundings. 

4.  Record  vessel  data  for  any  inbound  target(s),  and  include  the  time  you  recorded  the  data  for 
each  target: 


Initial  Bearing 

Target 

Course 

Speed 

CPA 

TCPA 

Time 

A 

B 

5.  Your  vessel  is  under  your  control:  what  color  lights  should  you  see  on  the  inbound  target 
when  you  emerge  from  under  the  Verrazano  Bridge,  at  about  1 1 13:00? 
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Figure  C-4.  Chartlet  of  a  portion  of  New  York  Harbor  upper  bay  for  use  in 

ARPA  exercise  E. 
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ARPA  Assessment  Exercise  F 


Name _ _ _ _  ARPA  Unit - 

Exercise  Time:  0500:00  Own  Ship  Course:  270°  Own  Ship  Speed:  20.0  knots 


Section  A 

1 .  Initialize  ARPA  in  the  display  presentation  and  mode  of  your  choice. 

2.  Set  safe  limits  CPA  to  1  nm. 

3.  Acquire  all  targets  in  the  order  that  will  give  you  the  best  information  about  a  potential 
collision. 

4.  Record  vessel  data,  and  include  the  time  you  recorded  the  data  for  each  target  (be  sure  to 
record  this  data  before  executing  any  course  change).  Record  data  for  targets  that  have 
highest  likelihood  of  collision  first. 


Initial  Bearing 

Target 

Course 

Speed 

CPA 

TCPA 

Time 

220 

A 

220 

B 

220 

C 

048 

D 

5.  What,  if  any,  problems  did  you  encounter  while  recording  the  above  data? 


6.  Assess  threat  and  maneuver  to  starboard  as  required.  Calculate  and  execute  course  change  at 
or  before  0510:00  for  the  maximum  CPA  allowable  given  the  circumstances.  Calculated 
course  change:  _ _ 

(Course  change  may  not  be  more  than  70°  to  starboard.) 
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ARP  A  Assessment  Exercise  F 


Name _ _ _  ARPA  Unit _ 

Section  B 

After  the  simulation  ends,  respond  to  the  following  multiple-choice  and  true/false  questions 
regarding  IMO  performance  standards  for  ARPA,  in  particular,  the  standards  relating  to  ARPA 
accuracy: 

1 .  ARP  As  with  auto-acquisition  capability: 

I.  Must  be  able  to  track  only  10  targets. 

II.  Must  have  a  means  to  cancel  targets. 

A.  I  only 

B.  II  only 

C.  Both  I  &  II 

D.  Neither  I  nor  II 

2.  (A)  typical  source(s)  of  error(s)  is/are: 

A.  Speed  log 

B.  Gyrocompass 

C.  Glint 

D.  All  of  the  above 

E.  None  of  the  above 

3.  The  primary  purpose  of  ARPA  is  to: 

A.  Give  more  accurate  CPAs 

B.  Reduce  the  workload  of  the  operator 

C.  Eliminate  the  requirement  to  plot 

D.  All  of  the  above 

E.  None  of  the  above 

4.  ARPA  with  auto-acquisition  should  have  a  facility  to  suppress  acquisition  in  certain  areas. 

A.  True 

B.  False 

5.  The  ARPA  should  be  capable  of  operating  with  a  true  motion  display  with  North-up  and 
either  course-up  azimuth  stabilization  or  head-up. 


A.  True 

B.  False 
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6.  One  minute  after  target  acquisition: 

I.  The  targets’  motion  trend  must  be  presented  by  the  ARPA. 
n.  The  targets’  predicted  motion  must  be  presented. 

A.  I  only 

B.  n  only 

C.  Bothl&n 

D.  Neither  I  nor  II 

7.  ARP  As  should  have  visual  and/or  audible  signals  for: 

I.  Targets  crossing  guard  zones 

n.  Lost  targets 

A.  I  only 

B.  nonly 

C.  Bothl&n 

D.  Neither  I  nor  n 

8.  All  ARPAs  must  present  course,  speed,  CPA,  TCPA,  range  and  bearing  of  at  least  20  tracked 
targets. 

A.  Tme 

B.  False 

9.  Target  swap  may  occur  when  two  or  more  targets  get  into  one  tracking  gate. 

A.  True 

B.  False 

10.  Exclusion  zones/areas  are  required  of  all  IMO-approved  ARPA  units. 

A.  True 

B.  False 

1 1.  ARPAs  should  give  the  operator  the  ability  to  activate  and  deactivate  operational  warnings. 

A.  True 

B.  False 

12.  All  ARPAs  are  required  to  display: 

I.  At  least  four  equally  spaced  past  positions 
n.  Target  history  for  at  least  eight  minutes 

A.  I  only 

B.  n  only 

C.  Both  I  and  D 

D.  Neither  I  nor  II 
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13.  All  IMO-approved  ARP  As  are  required  to  have  “nav-lines” 


A.  True 

B.  False 

14.  When  using  an  ARPA  in  the  course-up  mode: 

1.  Targets  smear  when  you  change  course. 

n.  The  heading  flasher  moves  in  accordance  with  the  amount  you  change  course. 

A.  I  only 

B.  n  only 

C.  Both  I  and  II 

D.  Neither  I  nor  n 

15.  At  least  eight  minutes  of  target  history  is  available  to  the  operator  as  soon  as  the  target  is 
acquired. 

A.  True 

B.  False 

16.  When  all  tracking  gates  are  full  the  computer  in  an  ARPA  must  determine  which  targets  to 
continue  to  track  and  what  target(s)  to  drop  when  a  new  target  appears.  The  process  the 
computer  goes  through  is  called: 

A.  Due  process 

B.  Prioritization 

C.  Rate-aiding 

D.  Glinting 

E.  Danger  assessment 

F.  All  of  the  above 

G.  None  of  these 

17.  The  center  of  the  ship  moves  over  the  full  length  of  the  ship  causing: 

A.  Seasickness 

B.  Spoking 

C.  Rate-aiding 

D.  Glinting 

18.  Setting  the  CPA  and  TCPA  safe  limits  to  zero  will  eliminate  close  quarters  situations  from 
developing. 

A.  True 

B.  False 
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19.  If  your  heading  flasher  is  not  running  through  a  collision  point,  there  is  no  risk  of  collision. 

A.  True 

B.  False 

20.  Which  of  the  following  are  optional? 

I.  Vectors 

n.  Graphic  representations 

A.  I  only 

B.  n  only 

C.  Both  I  and  n 

D.  Neither  I  nor  n 

21.  All  guard  zones  have  the  capability  of  auto-acquiring  targets. 

A.  True 

B.  False 


[This  page  intentionally  left  blank.] 


APPENDIX  D — Assessor  Instructions  for  ARPA  Assessment 


Appendix  D  provides  instructions  for  the  assessor  administrating  the  present  ARPA  assessment. 
For  each  of  the  six  exercises,  instructions  are  included  for  actions  that  the  assessor  is  required  to 
perform  before,  during,  and  after  each  simulation.  For  example,  to  assess  the  candidate’s 
knowledge  of  how  to  set  the  speed  and  compass  heading  in  exercise  A,  the  assessor  has  to  de¬ 
initialize  the  ARPA  units  before  candidates  arrive  so  that  the  speed  and  compass  settings  are 
incorrect. 

Instructions  are  also  provided  for  the  assessor  to  perform  specific  activities  during  the  simulation, 
such  as  disable  the  automatic  speed  log  in  exercise  C,  or  remove  a  target  from  the  candidates’ 
displays  in  exercise  A  to  set  off  the  lost  target  alarm.  The  assessor  is  also  required  to  watch  the 
instructor  console  to  monitor,  for  example,  that  candidates  maintain  at  least  the  minimum  CPA 
on  all  targets;  or  to  make  sure  that  candidates  have  manually  reset  the  speed  log  following  the 
automatic  speed  log  error.  Cueing  instructions  are  provided  for  activities  such  as  these. 

The  instructions  for  administering  the  written  test  questions  in  exercises  A,  C,  and  F  are  at  the 
end  of  the  assessor  instructions  for  those  exercises. 


Assessor  Instructions  for  ARPA  Assessment  Exercise  A 


IMPORTANT:  Instruct  candidates  to  read  all  instructions  before  each 

exercise  begins. 


1 .  Before  the  candidates  arrive,  de-initialize  ARPA  units  such  that  speed  and  compass  settings 
are  incorrect. 

2.  Before  getting  underway,  verify  that  each  candidate  has  correctly  initialized  his  or  her  ARPA 
unit: 

•  Speed  log  manually  set  to  20.0  knots. 

•  Compass  set  to  090°. 

•  Guard  zone  set  in  accordance  with  instructions. 

•  Exclusion  line  set  to  3  nm  north  of  own  ship. 

3.  Ask  each  candidate  to  demonstrate  the  various  display  presentations  and  display  modes  that 
are  available: 

•  North-up,  course-up  and  head-up  presentations. 

•  Relative  motion  and  true  motion  vector  modes. 

4.  After  demonstration,  verify  that  each  ARPA  unit  is  set  to  a  relative  motion,  stabilized  mode, 
and  that  automatic  acquisition  rings  have  been  turned  off. 
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5.  During  first  minute  of  simulation,  instruct  the  candidates  to  manually  acquire  all  targets  and 
to  record  vessel  data  at  the  appropriate  time. 

6.  Verify  that  each  candidate  has  correctly  manually  acquired  all  targets. 

7.  At  approximately  0106:00,  instruct  candidates  to  record  vessel  data  at  the  appropriate  time 
after  the  own  ship  maneuver. 

8.  At  01 13:00,  remove  target  C  from  each  candidate’s  display  to  actuate  the  lost  target  alarm. 

9.  Verify  that  each  candidate  correctly  responds  to  the  lost  target  alarm  by  reacquiring  target  C. 

10.  Beginning  approximately  01 15:00,  ask  candidates  to  demonstrate  the  ARPA’s  vessel  history 
capability. 

11.  After  simulation  ends,  administer  written  test  questions  on  vessel  history  patterns.  Distribute 
section  B  after  the  simulation  ends  and  allow  candidates  five  minutes  to  complete  the  written 
exam  questions. 

Assessor  Instructions  for  ARP  A  Assessment  Exercise  B 

1 .  Before  the  assessment  begins,  set  vector  scale  on  ARPA  units  to  3  minutes. 

2.  During  set-up,  tell  candidates  to  set  safe  limits  CPA  to  2  miles,  safe  limits  TCPA  to  24 
minutes,  and  vector  scale  to  6  minutes. 

3.  Ask  candidates  to  set  exclusion  area  to  exclude  southerly  fishing  vessels  (E,  F,  G,H,SlI)  and 
land. 

4.  Before  getting  underway,  instruct  candidates  to  draw  parallel  index  lines  to  maintain  a  3-mile 
distance  from  land  at  bearing  1 80°. 

5.  Verify  that  each  candidate  has  correctly  initialized  his  or  her  ARPA  unit: 

•  Speed  log  manually  set  to  20  knots. 

•  Compass  set  to  255°. 

•  Vectors  set  to  6  minutes. 

•  Guard  zone  set  to  3  nm. 

•  Exclusion  line  set  2.5  nm  to  south. 

•  Parallel  index  lines  drawn  in  accordance  with  instructions. 

6.  After  getting  underway,  tell  candidates  to  identify  and  record  the  code  for  any  racon(s)  that 
are  present  in  the  exercise. 

7.  At  approximately  1001:00,  ask  candidates  to  manually  acquire  all  targets  except  fishing 
vessels  to  the  north,  and  to  record  vessel  data  when  appropriate. 
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8.  Instruct  candidates  to  identify,  record  and  respond  to  any  warnings  that  occur  during  the 
exercise. 

9.  Throughout  the  exercise,  verify  that  each  candidate  maintains  ship’s  position  with  respect  to 
parallel  index  lines. 

10.  At  1018:00,  tell  candidates  to  change  course  to  270°,  keeping  the  point  of  land  at  3  nm,  in 
accordance  with  parallel  index  lines. 

1 1 .  After  own  ship  maneuver,  verify  that  each  candidate’s  ship  is  no  more  than  0.5  nm  on  either 
side  of  parallel  index  lines. 

Assessor  Instructions  for  ARP  A  Assessment  Exercise  C 

1.  Verify  that  each  candidate  has  correctly  initialized  his  or  her  ARPA  unit: 

•  Speed  log  manually  set  to  15.0  knots. 

•  Compass  set  to  180°., 

•  Vectors  set  to  true  motion. 

•  CPA  safe  limit  set  to  3  nm  and  TCPA  safe  limit  set  to  20  minutes. 

2.  After  getting  underway,  instruct  candidates  to  initialize  automatic  speed  log. 

3.  Tell  candidates  to  respond  to  any  operational  warnings  that  occur  during  the  exercise. 

4.  Instruct  candidates  to  acquire  all  targets  except  three  buoys  bearing  125°,  130°,  132°  true,  and 
to  record  vessel  data  when  appropriate. 

5.  Tell  candidates  to  record  which  vessels  have  potential  for  target  swap  at  0804:00. 

6.  Instruct  candidates  to  calculate  and  record  trial  maneuver,  and  to  execute  the  maneuver  by 
0808:00  so  that  resulting  CPA  is  3.0  or  greater. 

7.  Ask  candidates  to  record  vessel  data  when  appropriate  after  own  ship  changes  course. 

8.  At  console,  check  that  each  candidate  has  executed  a  large  enough  course  change  to  maintain 
a  minimum  CPA  of  3.0  nm  for  all  targets. 

9.  At  approximately  0812:00,  tell  candidates  not  to  change  course  again  until  after  0818:00  or 
later. 

10.  At  0814:00,  disable  speed  log,  and  then  verify  that  each  candidate  correctly  manually  resets 
his  or  her  speed  log. 

1 1.  At  approximately  0824:00,  verify  that  each  candidate  has  resumed  the  base  course  of  180°. 

12.  At  the  end  of  the  exercise,  ask  candidates  to  identify  any  targets  that  changed  course  or  speed 
during  the  exercise,  and  to  record  the  nature  and  direction  of  the  change  on  their  answer 
sheet. 
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13.  Administer  the  written  test  questions  regarding  precautions  to  be  taken  after  an  ARPA 
equipment  malfunction  occurs.  Distribute  section  B  after  the  simulation  ends  and  allow 
candidates  five  minutes  to  complete  the  written  test  questions. 

Assessor  Instructions  for  ARPA  Assessment  Exercise  D 

1 .  Verify  that  each  candidate  has  correctly  initialized  his  or  her  ARPA  unit: 

•  Speed  log  manually  set  to  15.0  knots. 

•  Compass  set  to  1 80°. 

•  Vectors  set  to  relative  motion. 

2.  Tell  candidates  to  acquire  all  targets. 

3.  Instruct  candidates  to  assess  and  record  the  COLREGS  situation  for  each  target. 

4.  Tell  candidates  to  record  vessel  data  as  soon  as  appropriate. 

5.  At  approximately  0605:00,  instruct  candidates  to  report  what  lights  they  see  on  each  target  at 
or  before  0610:00. 

6.  At  approximately  0613:00,  ask  candidates  to  identify  targets  that  have  changed  course  or 
speed,  and  record  the  nature  and  direction  of  the  change  on  their  answer  sheet. 

7.  At  0617:00,  tell  candidates  to  calculate,  report,  and  execute  a  course  change  as  soon  as 
possible  for  change  when  target  C  is  at  6  miles  for  a  3-mile  CPA. 

8.  Verify  that  each  candidate  resumes  the  base  course  of  180°  by  0638:00. 

Assessor  Instructions  for  ARPA  Assessment  Exercise  E 

1 .  Provide  candidates  with  chartlet  of  New  York  Harbor  Upper  Bay,  and  remind  them  that  no 
VHP  radio  or  pilot  is  available  during  the  exercise. 

2.  Verify  that  each  candidate  has  correctly  initialized  his  or  her  ARPA  unit: 

•  Speed  log  manually  set  to  15.0  knots. 

•  Compass  set  to  1 80°  true. 

3.  After  getting  underway,  verify  that  each  candidate  is  using  true  vectors,  and  that  candidates 
do  not  run  aground  (failure  is  automatic  for  collision  or  grounding). 

4.  Instruct  candidates  to  set  autodrift  (groundlock)  to  target  bearing  172°  at  4  nm  and  to 
manually  acquire  all  northbound  target(s). 

5.  Tell  candidates  to  report  what  lights  are  seen  on  inbound  target  from  approximately  1 1 13:00 
to  1115:00. 
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Assessor  Instructions  for  ARP  A  Assessment  Exercise  F 

1.  Verify  that  each  candidate  has  correctly  initialized  his  or  her  ARPA  unit; 

•  Speed  log  manually  set  to  20.0  knots. 

•  Compass  set  to  270°. 

•  CPA  safe  limits  set  to  1  nm. 

•  Display  presentation  and  vector  mode  settings  are  at  discretion  of  candidate. 

2.  Tell  candidates  to  respond  to  any  operational  warnings  that  occur  during  the  exercise. 

3.  Instruct  candidates  to  acquire  all  targets  and  record  vessel  data  as  soon  as  appropriate. 

4.  Ask  candidates  to  record  any  problems  they  encountered  in  reporting  vessel  data. 

5.  Instruct  candidates  to  assess  threat  and  maneuver  to  starboard  as  required,  and  to  calculate, 
record  and  execute  a  course  change  at  or  before  0510:00  for  the  maximum  CPA  allowable 
given  the  circumstances. 

6.  After  simulation  ends,  administer  section  B,  the  written  test  questions  regarding  IMO 
performance  standards  for  ARPA,  in  particular  the  standards  relating  to  ARPA  accuracy. 
Distribute  section  B  after  the  simulation  ends,  and  allow  candidates  30  minutes  to  complete 
the  written  questions. 
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